A method and device for human-machine autonomous collaboration based on fine-tuning a large model

Abstract

The application provides a man-machine autonomous cooperation method and device based on fine-tuning of a large model, wherein the method comprises the following steps: acquiring a part image and a target person action image; inputting the part image into a pre-trained first model to obtain a part recognition result; inputting the target person action image into a pre-trained second model to obtain an action classification result; fusing the part recognition result and the action classification result to form current work state information; inputting the current work state information into a large language model that is fine-tuned according to domain knowledge in advance; reasoning out a next assembly step based on a real-time assembly state, and decoding the next assembly step into a control instruction for controlling a cooperative mechanical arm to assist an operator to perform the next assembly step. Through the method, when man-machine autonomous cooperation is performed, only the part image and the target person action image need to be acquired, the next operation prompt can be actively given, the active understanding ability for the human state is stronger, and the understanding ability and autonomy of the machine in the man-machine cooperative assembly mode are improved.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to a human-computer autonomous collaboration method and device based on fine-tuning a large model. Background Technology

[0002] Human-machine collaboration (HRC) refers to the collaborative work and interaction between humans and robots, computers, or other intelligent systems, combining machine intelligence with human creativity and decision-making abilities. HRC assembly makes efficient and intelligent production of complex assembly tasks in the industrial field a reality. Human-machine collaborative state perception and joint decision-making are crucial foundations of HRC assembly and current research hotspots in the field. To achieve high-level collaboration between humans and machines, the key lies in the machine's ability to proactively perceive and understand human needs and states, thereby enabling efficient and safe cooperation. Advances in large language model technology have made proactive and autonomous HRC collaboration increasingly feasible. Therefore, it is necessary to research autonomous HRC assembly models.

[0003] With the development of artificial intelligence algorithms and the advancement of robotics, research on human-robot collaborative assembly has made some progress. In terms of human-robot collaborative state perception, WANG et al. used two AlexNets for human motion recognition and part / tool ​​recognition, respectively. AL-AMIN et al. used a personalized system based on a CNN classifier using skeleton data to recognize operator assembly actions, improving the accuracy of operator action recognition. Bilberg et al. used digital twin technology in human-robot collaborative models to extend the use of virtual simulation models to real-time control, skill-based dynamic task allocation between humans and robots, and robot program operation. Duan et al. proposed a human-robot collaborative framework for a dual-robot assembly system based on multimodal perception, providing multimodal information interaction for robot control through gesture perception, voice perception, human body perception, and visual perception. Dinges et al. used facial expressions trained and evaluated using AffectNet to predict operator fatigue levels in human-robot collaborative scenarios.

[0004] Current research on human-robot collaboration mainly focuses on command-based collaboration, where collaborative tasks are based on explicit instructions given by the operator. Exploration of more proactive and flexible collaboration models is still insufficient. Therefore, it is necessary to improve robots' ability to proactively understand human states. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a human-machine autonomous collaboration method, device and electronic device based on fine-tuning large model, so as to meet the needs of more proactive human-machine assistance and stronger understanding ability in the assembly process.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] According to a first aspect, embodiments of the present invention provide a human-machine autonomous collaboration method based on a fine-tuned large model, comprising: acquiring component images and target human action images; inputting the component images into a pre-trained first model to obtain component recognition results; inputting the target human action images into a pre-trained second model to obtain action classification results; fusing the component recognition results and action classification results to form current work status information; inputting the current work status information into a pre-fine-tuned large language model based on domain knowledge, inferring the next assembly step based on the real-time assembly status, and decoding it into control instructions to control a collaborative robotic arm to assist the operator in performing the next assembly step.

[0008] Optionally, the training process of the pre-trained first model includes: acquiring a labeled component image dataset; inputting the labeled component image dataset into the YOLO-V7 pre-trained model, wherein the backbone network of the YOLO-V7 pre-trained model that extracts general features is frozen; calculating the gradient of the head network parameters according to the first objective loss function, updating the head network parameters, until the component recognition rate of the first model reaches the preset accuracy.

[0009] Optionally, the training process of the pre-trained second model includes: acquiring multiple sets of target person action images, each set containing multiple temporally consecutive person action images; extracting multiple pose key points from each set of target person action images using a target image processing tool, and labeling the actions represented by the multiple pose key points in each set of person action images with corresponding action labels; converting the multiple pose key points in each set of target person action images into time series data, and inputting the time series data and corresponding action labels into the second model that integrates a long short-term memory network and an attention mechanism; in the second model, using an attention mechanism to generate corresponding weights for key actions, where each key action consists of multiple target pose key points; determining the difference between the predicted action classification result and the true label based on the second objective loss function, and using the gradient of the second objective loss function for backpropagation to update the network weights of the second model until the accuracy of the action classification result reaches the preset precision.

[0010] Optionally, the current work status information is input into a large language model that has been fine-tuned based on domain knowledge. The next assembly step is inferred based on the real-time assembly status, including: when the large language model receives the current work status information, it judges whether the target person's behavior represented by the component recognition result and action classification result complies with safety regulations according to the human-machine collaboration safety rules of the assembly domain; if it complies with safety regulations, it determines the current assembly progress represented by the current work status information and the next assembly step according to the assembly process document; and it formats the next assembly step according to the pre-input prompt word template to obtain the next assembly step instruction.

[0011] Optionally, the first objective loss function is:

[0012] L task =L cla +L CIoU ;

[0013] Among them, L cla Losses due to component identification and classification σ(x i ) is the Sigmoid function; y i This is a sign function; if the identified key component i belongs to the true category, it takes a value of 1; otherwise, it takes a value of 0. n represents the component category. CIoU For the loss of component position coordinates, A and B represent the coverage areas of the predicted bounding box and the ground truth bounding box, respectively; ρ is the diagonal distance between the minimum closure regions of the predicted bounding box and the ground truth bounding box; b is the center point of the predicted bounding box for component recognition; b gt α is the center point of the true bounding box; α is the weighting coefficient; v is used to measure the consistency of the aspect ratio, and its calculation formula is as follows: w and h are the width and height of the prediction box, respectively; w gt with h gt These represent the width and height of the actual bounding box, respectively.

[0014] Optionally, the second objective loss function is:

[0015]

[0016] Where M is the number of action categories; y ic The sign function is 1 if the true class of sample i is equal to c, and 0 otherwise; pi c This represents the predicted probability that observed sample i belongs to action category c.

[0017] Optionally, an attention mechanism is used to generate corresponding weights for key actions, including:

[0018]

[0019] Where, α ts Indicates the weight of key actions. h t It is the hidden state of the decoder, and is the hidden state of the encoder, W is the weight matrix, s′ represents the index from 1 to S, and S represents the length of the sequence.

[0020] Optionally, a human-machine autonomous collaboration method based on a fine-tuned large model further includes: inputting the next assembly step instruction into a digital twin model pre-established according to the real assembly scenario; the digital twin model adaptively grasps the corresponding parts according to the next assembly instruction to complete the assembly.

[0021] According to a second aspect, embodiments of the present invention provide a human-machine autonomous collaborative device based on a finely tuned large model, comprising: an image acquisition module for acquiring images of parts and images of target human actions; a parts recognition module for inputting the parts images into a pre-trained first model to obtain parts recognition results; an action classification module for inputting the target human action images into a pre-trained second model to obtain action classification results; a data fusion module for fusing the parts recognition results and action classification results to form current work status information; and an assembly instruction module for inputting the current work status information into a pre-fine-tuned large language model based on domain knowledge, inferring the next assembly step based on the real-time assembly status, and decoding it into control instructions to control a collaborative robotic arm to assist the operator in performing the next assembly step.

[0022] According to a third aspect, an embodiment of the present invention provides an electronic device, the device comprising: a memory, a processor, and a computer program stored thereon and executable on the processor, the processor executing the steps of the human-machine autonomous collaboration method based on a fine-tuned large model as described in the first aspect or any embodiment of the first aspect.

[0023] This embodiment provides a human-machine autonomous collaboration method based on a fine-tuned large model. It uses OpenAI ChatGPT as the foundation of a large language model, fine-tuning it to provide prompts for the next step based on component recognition and action classification results. This allows the collaborative robotic arm to execute the next assembly step. In this embodiment, component recognition and action classification results are combined to provide richer input information to the large language model, thereby improving the accuracy of understanding the current assembly progress. Furthermore, only component images and target human action images are needed to proactively provide prompts for the next step, enhancing autonomy. Therefore, the method proposed in this embodiment has a stronger proactive understanding of human states, promoting the realization of human-machine autonomous collaboration and improving the machine's understanding and autonomy in human-machine collaborative assembly modes.

[0024] This embodiment provides a human-computer autonomous collaboration method based on fine-tuning a large model. By using transfer learning, the backbone network that extracts general features from the YOLO-V7 pre-trained model is frozen, and only the parameters of the head network used for classification are fine-tuned. This can accelerate the convergence speed and reduce computational resources when training data is insufficient.

[0025] This embodiment provides a human-computer autonomous collaboration method based on fine-tuning a large model. Since action recognition relies on time series for judgment, this embodiment uses a long short-term memory network and an attention mechanism to enable the model to accurately capture information in the time series. By applying an attention mechanism to key actions and generating corresponding weights, the model can more accurately recognize and understand actions. The method proposed in this embodiment can improve the discrimination accuracy of action classification results.

[0026] Other advantages, objectives, and features of the invention will be set forth in the following description and will be apparent to those skilled in the art in some respects, or may be learned by practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0027] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following figures are provided for illustration:

[0028] Figure 1 This is a flowchart illustrating a specific example of a human-computer autonomous collaboration method based on a fine-tuned large model according to the present invention.

[0029] Figure 2 This is a diagram illustrating the implementation effect of component identification and action classification results in this invention.

[0030] Figure 3 This is a schematic diagram illustrating the method of constructing a labeled component image dataset for reducer assembly in this invention;

[0031] Figure 4 This is a diagram showing the training results of transfer learning on the YOLO-V7 pre-trained model in this invention;

[0032] Figure 5 This is a schematic diagram of the training sample data structure during the training of the second model in this invention;

[0033] Figure 6 This is a schematic diagram illustrating the process of classifying operator actions using the second model that integrates LSTM and attention mechanisms in this invention.

[0034] Figure 7This is a schematic diagram illustrating the use of transfer learning to train the YOLO-V7 pre-trained model in this invention, the use of the trained model to identify parts, and the use of a long short-term memory network to classify the operator's actions.

[0035] Figure 8 , 9 This is a schematic diagram of the process by which the finely adjusted large language model in this invention obtains the current assembly progress and the instructions for the next assembly step based on the component recognition results and action classification results.

[0036] Figure 10 This is a schematic diagram of a human-machine autonomous collaborative assembly scenario driven by the fusion of a large language model and a digital twin, which is applied in this invention.

[0037] Figure 11 This is a schematic diagram of how the digital twin module receives the next assembly instruction, which is output in the large language model based on the component identification results and action classification results in this invention.

[0038] Figure 12 This is a schematic block diagram of a specific example of an electronic device in an embodiment of the present invention. Detailed Implementation

[0039] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can also refer to the internal connection of two components; and they can refer to a wireless connection or a wired connection. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0041] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0042] This invention provides a human-computer autonomous collaboration method based on fine-tuning a large model, such as... Figure 1 As shown, it includes:

[0043] S101, acquire component images and target character motion images;

[0044] S103, Input the component image into the pre-trained first model to obtain the component recognition result;

[0045] S105, Input the target person's action image into the pre-trained second model to obtain the action classification result;

[0046] S107, integrates component identification results and motion classification results to form current operation status information;

[0047] S109: Input the current work status information into the large language model that has been fine-tuned in advance based on domain knowledge, infer the next assembly step based on the real-time assembly status, and decode it into control instructions to control the collaborative robotic arm to assist the operator in performing the next assembly step.

[0048] For example, the method of acquiring images of parts and target personnel can be by capturing images of target personnel and parts using a camera. The target personnel can be a parts operator. The pre-trained first model can be a region-based convolutional neural network (Faster R-CNN), which uses a region proposal network to quickly and efficiently locate and identify objects in the image; the pre-trained first model can also be an Inception network model, which processes images in parallel using convolutional kernels of different sizes to capture features at different scales; or the pre-trained first model can be a YOLO model, which can quickly and accurately identify multiple objects in an image, suitable for parts recognition scenarios requiring rapid response. This embodiment does not limit the first model, and those skilled in the art can determine it as needed.

[0049] The pre-trained second model can be a Long Short-Term Memory (LSTM) network, which is better able to capture long-term dependencies and is suitable for processing action recognition over long time sequences. Alternatively, the pre-trained second model can be a dual-stream network, which combines spatial flow (processing single-frame images) and temporal flow (processing optical flow images) to simultaneously capture spatial and temporal features for action recognition. This embodiment does not limit the second model; those skilled in the art can determine it as needed.

[0050] When training the second model for action classification and recognition, multiple temporally consecutive images of target person actions can be labeled with corresponding action tags. These action tags can be three state data points of a person in a human-computer collaboration scenario: assembling, waiting, and leaving, to facilitate supervised training. The completed training of the second model will result in the action classification of the target person action images as any one of assembling, waiting, or leaving.

[0051] like Figure 2As shown in the figure, this embodiment provides the implementation effect diagram of the component recognition result and the action classification result. The left side is the action recognition module, and the upper left corner shows the operator action recognition output result. The right side is the key component recognition module, and the bottom shows the type and quantity of the key components identified.

[0052] Reconstructing a domain-specific high-performance large language model requires a large amount of data and computing resources, which is detrimental to the rapid deployment of intelligent collaborative robotic arms. Furthermore, general-purpose large language models already contain rich knowledge and semantic understanding capabilities. Therefore, this invention proposes a method of fine-tuning a general-purpose large language model based on assembly domain knowledge to meet the need for prompting key information in human-machine collaboration.

[0053] The large language model (LLM) pre-calibrated based on assembly domain data can be fine-tuned using GPT3.5 based on the LangChain framework. Three logical prompt chains are used to fine-tune the large language model: human-computer collaboration safety rules, assembly rules, and output command rules. The assembly rules can be fine-tuned by adding assembly process files to the model, while the output command rules can be fine-tuned by pre-inputting prompt word templates. During output, the content to be output is matched with the prompt word templates to output the specified prompt words.

[0054] After fine-tuning the large language model, it needs to be integrated with the collaborative robotic arm to form an intelligent agent. First, the component recognition results and action classification results are fused to form the current work status information. This work status information is used as input, and the work environment information is encoded and organized into a standardized structure for input into the large language model. Then, the large language model in the intelligent collaborative robotic arm parses the input information, outputting the current assembly progress and the next assembly step, and generating prompt text. For example, when it recognizes that the machine body, large gear, and gear shaft are currently being assembled, and the component operator's status is "assembling," the model outputs the prompt text: "Lid needs to be placed; perform the lid-grabbing operation." The output prompt text is decoded into commands to control the robotic arm, instructing it to perform the next collaborative task. It is worth noting that if an abnormal command is encountered during the task, the collaborative task is stopped to achieve a proactive and safe human-machine collaboration mode.

[0055] This embodiment provides a human-machine autonomous collaboration method based on a fine-tuned large model. It uses OpenAI ChatGPT as the foundation of a large language model, fine-tuning it to provide prompts for the next step based on component recognition and action classification results. This allows the collaborative robotic arm to execute the next assembly step. In this embodiment, component recognition and action classification results are combined to provide richer input information to the large language model, thereby improving the accuracy of understanding the current assembly progress. Furthermore, only component images and target human action images are needed to proactively provide prompts for the next step, enhancing autonomy. Therefore, the method proposed in this embodiment has a stronger proactive understanding of human states, promoting the realization of human-machine autonomous collaboration and improving the machine's understanding and autonomy in human-machine collaborative assembly modes.

[0056] As an optional implementation, the training process of the pre-trained first model includes:

[0057] Obtain a tagged dataset of component images;

[0058] The labeled component image dataset is input into the YOLO-V7 pre-trained model, where the backbone network of the YOLO-V7 pre-trained model that extracts general features is frozen;

[0059] The gradient of the head network parameters is calculated based on the first objective loss function, and the head network parameters are updated until the component recognition rate of the first model reaches the preset accuracy.

[0060] For example, the first model was trained in a Python 3.7 and PyTorch environment, using an Intel Core™ i9-12900H and an NVIDIA GeForce 3070Ti device.

[0061] The method for obtaining labeled component image datasets, taking the reducer assembly case as an example, is as follows: Figure 3 As shown, this is a dataset of 2601 component images of a reducer model. The dataset is labeled, the main components are labeled, and the coordinates and other information of the components on the images are normalized to form a labeled component image dataset.

[0062] The component identification process mainly relies on the backbone and head networks of the YOLO-V7 pre-trained model for feature extraction and classification. However, the initial performance of the YOLO-V7 model during training is often unsatisfactory, with a large loss function and significant differences in training results. Transfer learning can accelerate convergence and reduce computational resources when training data is insufficient. Therefore, the YOLO-V7 pre-trained model is used for transfer learning to effectively utilize the general weights for identifying similar features of components. During training, the weights of the first 50 layers of the backbone network are frozen, and only the 50 layers of the head network are trained. The batch size is 8, the image size is set to 640*640, and the learning rate is set to 0.01. After 248 epochs of training, the results are as follows. Figure 4 As shown, the backbone network for extracting general features is frozen, and only the parameters of the head network used for classification are fine-tuned, enabling the model to converge quickly.

[0063] The parameters of the head network used for classification can be fine-tuned by using gradient descent to minimize the first objective loss function to update the parameters. When the component recognition rate of the first model reaches the preset accuracy, the update stops and the training of the first model is completed. The preset accuracy can be 95%. This embodiment does not limit the preset accuracy, but those skilled in the art can determine it as needed.

[0064] Since the accuracy of the component identification result in this embodiment includes two aspects: first, whether the coordinates of the component positioning are accurate, and second, whether the component identification result is accurate, as a preferred implementation, the first target loss function can be determined based on the coordinate loss of component positioning and the component identification classification loss. The setting of the coordinate loss of component positioning and the component identification classification loss is not limited in the art, and those skilled in the art can determine it as needed.

[0065] This embodiment provides a human-computer autonomous collaboration method based on fine-tuning a large model. By using transfer learning, the backbone network that extracts general features from the YOLO-V7 pre-trained model is frozen, and only the parameters of the head network used for classification are fine-tuned. This can accelerate the convergence speed and reduce computational resources when training data is insufficient.

[0066] As an optional implementation, the pre-trained second model training process includes:

[0067] Acquire multiple sets of target character motion images, each set containing multiple time-series consecutive motion images of the character;

[0068] The target image processing tool extracts multiple pose key points from each set of target human action images, and assigns corresponding action labels to the actions represented by the multiple pose key points in each set of human action images.

[0069] Multiple pose key points in each group of target person action images are converted into time series data. The time series data and corresponding action labels are input into a second model that integrates a long short-term memory network and an attention mechanism. In the second model, an attention mechanism is used to generate corresponding weights for key actions. Key actions are composed of multiple target pose key points.

[0070] Based on the second objective loss function, the difference between the predicted action classification result and the true label is determined, and the gradient of the second objective loss function is used for backpropagation to update the network weights of the second model until the accuracy of the action classification result reaches the preset precision.

[0071] For example, the second model was trained in a Python 3.7 and PyTorch environment, using an Intel Core™ i9-12900H and an NVIDIA GeForce 3070Ti device.

[0072] One way to obtain training samples is to capture 30 videos of each action of the target person, each consisting of 30 frames. The images of the target person's actions contained in each 30-frame video are considered a group, and the data structure is as follows: Figure 5 As shown, the MEDIAPIPE image processing framework was used to extract key points of the operator's posture in each group of target human motion images, reducing the interference of complex environments on operator motion recognition. The extracted key points of operator posture include 21 hand key points, 33 posture skeleton key points, and 468 facial key points. These key points represent the operator's motion details, torso information, and facial position and orientation, respectively. These key points were then labeled with action tags, which could include assembly, waiting, and leaving.

[0073] Long Short-Term Memory (LSTM) networks exhibit unique advantages when processing time-series information such as assembly line movements. LSTMs can "remember" the historical information of an operator's actions, incorporating previous context when processing the current action, enabling them to capture temporal dependencies in the action sequence. The activation functions tanh and σ, along with the LSTM's input, output, and forget gate structures, work together to allow the model to accurately capture information within the action sequence. Therefore, this embodiment of the invention constructs a three-layer LSTM to deeply encode the input pose keypoint sequence. Each LSTM layer employs the ReLU activation function and uses the adam optimizer. After 200 epochs of training, the neural network essentially converges and achieves accurate classification results.

[0074] In practical applications, certain action features of operators are of high importance, such as the operator's elbow flexion movement. To focus more on certain key action information, an attention mechanism is added to the model to dynamically generate weight vectors and weight the input data. Key actions consist of multiple target pose key points.

[0075] The second objective loss function can be the cross-entropy function, which monitors the accuracy of action classification in real time, determines the difference between the predicted action classification result and the true label, and uses the gradient of the second objective loss function for backpropagation to update the network weights of the second model until the accuracy of the action classification result reaches the preset accuracy, which can be 95%. This embodiment does not limit the preset accuracy, and those skilled in the art can determine it as needed.

[0076] The second model, which integrates LSTM and attention mechanisms, performs the following process for classifying operator actions: To understand the operator's working status in real time and accurately capture and understand their actions, images of the operator are first captured. Key body information is extracted and converted into time-series data. Finally, this data is input into a neural network integrating three layers of LSTM and attention mechanisms for accurate prediction of action classification. Specifically... Figure 6 As shown.

[0077] Based on the pre-trained first model in the above embodiments, the component recognition process and action classification process of this method are as follows: Figure 7 As shown, transfer learning is used to train the YOLO-V7 pre-trained model, the trained model is used to identify parts, and a long short-term memory network is used to classify the operator's actions.

[0078] This embodiment provides a human-computer autonomous collaboration method based on fine-tuning a large model. Since action recognition relies on time series for judgment, this embodiment uses a long short-term memory network and an attention mechanism to enable the model to accurately capture information in the time series. By applying an attention mechanism to key actions and generating corresponding weights, the model can more accurately recognize and understand actions. The method proposed in this embodiment can improve the discrimination accuracy of action classification results.

[0079] As an optional implementation, the current work status information is input into a large language model that has been pre-tuned based on domain knowledge, and the next assembly step is inferred based on the real-time assembly status, including:

[0080] When the large language model receives the current job status information, it judges whether the target person's behavior represented by the component recognition results and action classification results complies with safety regulations based on the human-machine collaboration safety rules in the assembly field.

[0081] If safety regulations are met, the current assembly progress and the next assembly step are determined according to the assembly process documents.

[0082] Based on the pre-entered prompt template, the next assembly step is formatted to obtain the next assembly step instruction.

[0083] For example, such as Figure 8 As shown, this includes the process by which the finely tuned large language model obtains the current assembly progress and the next assembly step instructions based on the current job status information. When the large language model receives the current job status information, it first integrates the component recognition results and action classification results to form a comprehensive contextual understanding. Then, it analyzes the received action classification results and components to determine the specific operation steps and actions performed by the operator, such as... Figure 9 The operator shown is performing action n, assembling component n. The identified behavior is matched against rules in the human-machine collaboration safety rule base to determine if it meets the prescribed safety standards. If it does, the current assembly progress and the next assembly step are determined based on the assembly process document, according to the component identification results and action classification results. The current assembly progress and the next assembly step are formatted according to a pre-input prompt word template to obtain the prompt word for the current assembly progress and the instruction for the next assembly step, such as... Figure 9 As shown, the output indicates that the operator is performing an assembly step: current state n, and the robotic arm needs to perform the following operation: next assembly step instruction n. The display panel showing the output can display only the current state and the next assembly step instruction. This formatted output method enables the output of formatted human-machine collaboration instructions, thus simplifying the encoding and decoding process.

[0084] As an optional implementation method, the first objective loss function is:

[0085] L task =L cla +L CIoU ;

[0086] Among them, L cla Losses due to component identification and classification σ(x i ) is the Sigmoid function; y i This is a sign function; if the identified key component i belongs to the true category, it takes a value of 1; otherwise, it takes a value of 0. n represents the component category. CIoU For the loss of component position coordinates, A and B represent the coverage areas of the predicted bounding box and the ground truth bounding box, respectively; ρ is the diagonal distance between the minimum closure regions of the predicted bounding box and the ground truth bounding box; b is the center point of the predicted bounding box for component recognition; b gt α is the center point of the true bounding box; α is the weighting coefficient; v is used to measure the consistency of the aspect ratio, and its calculation formula is as follows: w and h are the width and height of the prediction box, respectively; w gt with h gt These represent the width and height of the actual bounding box, respectively.

[0087] This embodiment provides a human-computer autonomous collaboration method based on fine-tuning a large model, which considers the loss of component recognition and classification and the loss of component position coordinates, thereby improving the accuracy of component recognition results of the trained first model.

[0088] As an optional implementation, the second objective loss function is:

[0089]

[0090] Where M is the number of action categories; yi c The sign function is 1 if the true class of sample i is equal to c, and 0 otherwise; p ic This represents the predicted probability that observed sample i belongs to action category c.

[0091] As an optional implementation, an attention mechanism is used to generate corresponding weights for key actions, including:

[0092]

[0093] Where, α ts Indicates the weight of key actions. h t It is the hidden state of the decoder, and is the hidden state of the encoder, W is the weight matrix, s′ represents the index from 1 to S, and S represents the length of the sequence.

[0094] This embodiment provides a human-computer autonomous collaboration method based on fine-tuning a large model. By employing an attention mechanism for key actions and generating corresponding weights, the model can more accurately identify and understand actions, helping it to focus on processing the most important information, thereby improving computational efficiency and reducing the demand for computing resources.

[0095] As an optional implementation method, a human-computer autonomous collaboration method based on fine-tuning a large model further includes:

[0096] Input the instructions for the next assembly step into the digital twin model that has been pre-built based on the real assembly scenario;

[0097] The digital twin model adaptively retrieves the corresponding parts based on the next assembly instruction and completes the assembly.

[0098] For example, such as Figure 10 As shown, the method of this embodiment is applied to a human-machine autonomous collaborative assembly scenario driven by the fusion of a large language model and a digital twin. In this scenario, the human-machine collaborative state perception module executes the component identification results and action classification results, and uses the component identification results and action classification results output by the human-machine collaborative state perception module as input to the large language model to obtain the next assembly instruction. In this scenario, this embodiment proposes a human-machine collaborative digital twin scenario, and uses this digital twin scenario as the underlying driver to support the human-machine collaborative state perception module and the large language model. The digital twin module simulates the assembly process, adaptively responds to the next assembly instruction, and completes the next assembly task.

[0099] A digital twin model is pre-built based on a real assembly scenario. The creation process can include: Digital twin technology is a method of creating virtual representations of physical entities or processes, allowing for the simulation, analysis, and optimization of various aspects of a system. When constructing the assembly process of an assembly plant, digital twins can provide the following: Creating 3D models of assembly lines and robotic arms using CAD (Computer-Aided Design) software, including machines, parts, conveyors, and other equipment. Simulating the robotic arm assembly process using simulation software, including the use of parts and the operation of equipment. Integrating data from sensors, equipment, and production management systems to update the digital twin model in real time, ensuring it reflects the actual state of the assembly workshop.

[0100] The finely tuned large language model can quickly analyze and issue the next task instruction based on the key components and the operator's current actions, and guide the robotic arm on how to cooperate effectively, such as... Figure 11 As shown, in this embodiment, the digital twin module receives these instructions and adaptively retrieves the required components according to the instructions, thus completing the effectiveness verification of the method.

[0101] This embodiment provides a human-computer autonomous collaboration device based on a fine-tuned large model, including:

[0102] The image acquisition module is used to acquire images of parts and images of the target person's movements;

[0103] The component recognition module is used to input component images into a pre-trained first model to obtain component recognition results.

[0104] The action classification module is used to input the target person's action image into a pre-trained second model to obtain the action classification result;

[0105] The data fusion module is used to combine the component identification results and action classification results to form the current operation status information;

[0106] The assembly instruction module inputs the current operation status information into a large language model that has been fine-tuned based on domain knowledge. Based on the real-time assembly status, it infers the next assembly step and decodes it into control instructions to control the collaborative robotic arm to assist the operator in performing the next assembly step.

[0107] This application also provides an electronic device, such as... Figure 12 As shown, processor 501 and memory 502 are connected via a bus or other means.

[0108] Processor 501 can be a central processing unit (CPU). Processor 501 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0109] The memory 502, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the human-machine autonomous collaboration method based on fine-tuning a large model in this embodiment of the invention. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory.

[0110] Memory 502 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 502 may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0111] The one or more modules are stored in the memory 502, and when executed by the processor 501, they perform actions such as... Figure 1 The embodiment shown is a human-computer autonomous collaboration method based on fine-tuning a large model.

[0112] For specific details regarding the aforementioned drone equipment, please refer to the relevant documentation. Figure 1 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here.

[0113] This embodiment also provides a computer storage medium storing computer-executable instructions that can execute the human-machine autonomous collaboration method in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0114] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A human-computer autonomous collaboration method based on fine-tuning a large model, characterized in that, include: Acquire images of component parts and images of the target person's movements; The component images are input into a pre-trained first model to obtain component recognition results; The target person's action image is input into a pre-trained second model to obtain the action classification result; The component identification results and motion classification results are combined to form the current operation status information; The current work status information is input into a large language model that has been fine-tuned based on domain knowledge. The next assembly step is inferred based on the real-time assembly status and decoded into control instructions to control the collaborative robotic arm to assist the operator in performing the next assembly step. The training process of the pre-trained first model includes: Obtain a tagged dataset of component images; The labeled component image dataset is input into the YOLO-V7 pre-trained model, where the backbone network of the YOLO-V7 pre-trained model that extracts general features is frozen; The gradient of the head network parameters is calculated based on the first objective loss function, and the head network parameters are updated until the component recognition rate of the first model reaches the preset accuracy. The training process for the pre-trained second model includes: Acquire multiple sets of target character motion images, each set containing multiple time-series consecutive motion images of the character; The target image processing tool extracts multiple pose key points from each set of target human action images, and assigns corresponding action labels to the actions represented by the multiple pose key points in each set of human action images. Multiple pose key points in each group of target person action images are converted into time series data. The time series data and corresponding action labels are input into a second model that integrates a long short-term memory network and an attention mechanism. In the second model, an attention mechanism is used to generate corresponding weights for key actions. Key actions are composed of multiple target pose key points. Based on the second objective loss function, the difference between the predicted action classification result and the true label is determined, and the gradient of the second objective loss function is used for backpropagation to update the network weights of the second model until the accuracy of the action classification result reaches the preset precision. The first objective loss function is: ； in, Losses due to component identification and classification , For the Sigmoid function; For symbolic functions, if the key components are identified If it belongs to the true category, the value is 1; otherwise, it is 0. n represents the component category. For the loss of component position coordinates, , A and B are the coverage areas of the predicted bounding box and the ground truth bounding box, respectively; This is the diagonal distance between the minimum closure regions of the predicted bounding box and the ground truth bounding box; Identify the center point of the predicted bounding box for the component; The center point of the true bounding box; These are the weighting coefficients; The formula used to measure the consistency of aspect ratio is as follows: , and These are the width and height of the prediction box, respectively; and These represent the width and height of the actual bounding box, respectively.

2. The human-computer autonomous collaboration method based on a fine-tuned large model according to claim 1, characterized in that, The current work status information is input into a pre-tuned large language model based on domain knowledge. The next assembly step is inferred based on the real-time assembly status, including: When the large language model receives the current job status information, it judges whether the target person's behavior represented by the component recognition results and action classification results complies with safety regulations based on the human-machine collaboration safety rules in the assembly field. If safety regulations are met, the current assembly progress and the next assembly step are determined according to the assembly process documents. Based on the pre-entered prompt template, the next assembly step is formatted to obtain the next assembly step instruction.

3. The human-computer autonomous collaboration method based on a fine-tuned large model according to claim 1, characterized in that, The second objective loss function is: ； in, The number of action categories; The sign function is set to 1 if the true class of sample i is equal to c, and 0 otherwise. This represents the predicted probability that observed sample i belongs to action category c.

4. The human-computer autonomous collaboration method based on a fine-tuned large model according to claim 1, characterized in that, An attention mechanism is used to generate corresponding weights for key actions, including: ; in, Indicates the weight of key actions. , It is the hidden state of the decoder, and These are the hidden state of the encoder, and W is the weight matrix. Indicates from 1 to The index of the sequence is S, where S represents the length of the sequence.

5. The human-computer autonomous collaboration method based on a fine-tuned large model according to claim 1, characterized in that, Also includes: Input the instructions for the next assembly step into the digital twin model that has been pre-built based on the real assembly scenario; The digital twin model adaptively retrieves the corresponding parts based on the next assembly instruction and completes the assembly.

6. A human-machine autonomous collaborative device based on a fine-tuned large model, characterized in that, include: The image acquisition module is used to acquire images of parts and images of the target person's movements; The component recognition module is used to input component images into a pre-trained first model to obtain component recognition results. The action classification module is used to input the target person's action image into a pre-trained second model to obtain the action classification result; The data fusion module is used to combine the component identification results and action classification results to form the current operation status information; The assembly instruction module inputs the current work status information into a large language model that has been fine-tuned based on domain knowledge. It infers the next assembly step based on the real-time assembly status and decodes it into control instructions to control the collaborative robotic arm to assist the operator in performing the next assembly step. The training process of the pre-trained first model includes: Obtain a tagged dataset of component images; The labeled component image dataset is input into the YOLO-V7 pre-trained model, where the backbone network of the YOLO-V7 pre-trained model that extracts general features is frozen; The gradient of the head network parameters is calculated based on the first objective loss function, and the head network parameters are updated until the component recognition rate of the first model reaches the preset accuracy. The training process for the pre-trained second model includes: Acquire multiple sets of target character motion images, each set containing multiple time-series consecutive motion images of the character; The target image processing tool extracts multiple pose key points from each set of target human action images, and assigns corresponding action labels to the actions represented by the multiple pose key points in each set of human action images. Multiple pose key points in each group of target person action images are converted into time series data. The time series data and corresponding action labels are input into a second model that integrates a long short-term memory network and an attention mechanism. In the second model, an attention mechanism is used to generate corresponding weights for key actions. Key actions are composed of multiple target pose key points. Based on the second objective loss function, the difference between the predicted action classification result and the true label is determined, and the gradient of the second objective loss function is used for backpropagation to update the network weights of the second model until the accuracy of the action classification result reaches the preset precision. The first objective loss function is: ； in, Losses due to component identification and classification , For the Sigmoid function; For symbolic functions, if the key components are identified If it belongs to the true category, the value is 1; otherwise, it is 0. n represents the component category. For the loss of component position coordinates, , A and B are the coverage areas of the predicted bounding box and the ground truth bounding box, respectively; This is the diagonal distance between the minimum closure regions of the predicted bounding box and the ground truth bounding box; Identify the center point of the predicted bounding box for the component; The center point of the true bounding box; These are the weighting coefficients; The formula used to measure the consistency of aspect ratio is as follows: , and These are the width and height of the prediction box, respectively; and These represent the width and height of the actual bounding box, respectively.

7. An electronic device, the device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor performs the steps of a human-computer autonomous collaboration method based on a fine-tuned large model as described in any one of claims 1-5.

Images