Man-machine cooperation data set construction and anomaly detection method fusing multi-modal large model

Abstract

The invention discloses a man-machine cooperation data set construction and anomaly detection method fusing a multi-modal large model, and relates to the technical field of man-machine cooperation, and the method comprises the steps: controlling a virtual digital human and a virtual mechanical arm to carry out virtual man-machine cooperation in a virtual man-machine cooperation scene based on a man-machine motion label, in the process of virtual man-machine cooperation, human body collision parts, multi-angle man-machine scene images and joint posture data are obtained, timestamp alignment is further carried out, a multi-modal data set is formed, the multi-modal data set is converted into a vision-language instruction data set through a multi-modal large model, and the vision-language instruction data set is used as a virtual man-machine cooperation model. And utilizing the vision-language instruction data set to carry out fine tuning on the multi-modal large model, and utilizing the multi-modal large model after fine tuning to realize man-machine cooperation anomaly detection in the virtual man-machine cooperation scene. According to the method, a high-quality multi-modal data set can be efficiently constructed, fine tuning training can be carried out on a multi-modal large model, and man-machine cooperation anomaly detection in a virtual man-machine cooperation scene is realized.

Description

Technical Field

[0001] The present application relates to the field of human-machine collaboration technology, and in particular to a method for constructing a human-machine collaboration data set and detecting anomalies by integrating a multimodal large model. Background Art

[0002] As robotics technology continues to mature and artificial intelligence develops rapidly, human-machine collaboration technology, as a cutting-edge industrial production paradigm, is gradually changing the face of traditional manufacturing. In the current field of human-machine collaboration, scene anomaly detection (i.e., anomaly detection in human-machine collaboration scenarios) technology is not only the key to ensuring production safety and efficiency, but also the core driving force for promoting continuous progress in the field of human-machine collaboration. Traditional scene anomaly detection technology is based on pure visual models, which often rely on single image information and are difficult to fully capture multimodal information in complex working environments. The multimodal large model (also called the multimodal large language model) can be fine-tuned for specific scene tasks through high-quality human-machine collaboration data sets, bringing a new solution to scene anomaly detection.

[0003] However, the collection of human-machine collaboration scene data in current human-machine collaboration datasets faces many challenges. First, the debugging of complex hardware equipment is time-consuming and labor-intensive, resulting in a long and inefficient data collection cycle, and the need to manually label the data, which is inefficient; second, the potential collision risk in human-machine collaboration scenarios will not only cause data collection interruptions or damage to equipment, but may also threaten the personal safety of operators, further affecting the integrity and availability of the dataset; finally, in actual scenarios, due to space limitations and viewing angle issues, the collected single-angle images lack global information about the scene. The above-mentioned problems jointly limit the quality and efficiency of human-machine collaboration datasets.

[0004] The concept of multimodality in human-machine collaboration scenarios is not limited to traditional sensory categories such as vision and hearing in a narrow sense, but has expanded to a wider range of information dimensions, including but not limited to human body motion posture, robot arm joint data, work scene images and other types. Integrating the above modal information of different dimensions to generate a high-quality multimodal data set based on human-machine collaboration scenarios can meet the specific downstream task fine-tuning training of existing multimodal large models, not only for anomaly detection for a variety of visual tasks, but also for zero-sample understanding and reasoning of untrained complex multivariate scenarios with the significant generalization and emergence capabilities of multimodal large models, bringing more dimensions and deeper analysis to anomaly detection in human-machine collaboration scenarios, and providing more intelligent technical support for the optimization and upgrading of human-machine collaboration technology.

[0005] Based on this, this field urgently needs an efficient method for constructing high-quality multimodal datasets. Summary of the invention

[0006] The purpose of this application is to provide a method for constructing a human-computer collaboration dataset and anomaly detection that integrates a multimodal large model, which can efficiently construct a high-quality multimodal dataset and further fine-tune the multimodal large model to achieve human-computer collaboration anomaly detection in a virtual human-computer collaboration scenario.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a method for constructing a human-machine collaborative dataset and detecting anomalies by integrating a multimodal large model. The method for constructing a human-machine collaborative dataset and detecting anomalies by integrating a multimodal large model comprises:

[0009] Control virtual digital humans and virtual robotic arms to perform virtual human-machine collaboration in virtual human-machine collaboration scenarios based on human-machine action labels;

[0010] In the process of virtual human-machine collaboration, the human body collision part when the virtual digital human collides with the physical assets in the virtual human-machine collaboration scene, the multi-angle human-machine scene images of the virtual human-machine collaboration scene and the joint posture data of the virtual robotic arm are obtained;

[0011] Performing timestamp alignment on the human body collision part, the multi-angle human-machine scene image and the joint posture data, and composing the human body collision part, the multi-angle human-machine scene image and the joint posture data after the timestamp alignment into human-machine collaborative virtual scene time series data corresponding to the human-machine action label;

[0012] Combining each of the human-machine action labels and the human-machine collaborative virtual scene time series data corresponding to the human-machine action labels into a multimodal data set;

[0013] Converting the multimodal dataset into a visual-language instruction dataset using a multimodal macro model;

[0014] The multimodal large model is fine-tuned using the visual-language instruction data set to obtain a fine-tuned multimodal large model; and the fine-tuned multimodal large model is used to implement human-machine collaboration anomaly detection in a virtual human-machine collaboration scenario.

[0015] Optionally, the human-machine action label includes: a label of the tester's body posture, a label of the robot arm's working status and a label of the human-machine space distance; the label of the tester's body posture includes standing, squatting, sitting and lying down; the label of the robot arm's working status includes still, moving but not grasping and grasping; the label of the human-machine space distance includes safe and normal, warning reminder and dangerous braking.

[0016] Optionally, controlling the virtual digital human and the virtual robotic arm to perform virtual human-machine collaboration in a virtual human-machine collaboration scene based on the human-machine action label specifically includes:

[0017] Acquire human body posture data of a real tester and joint motion data of a real robot arm; the human body posture data and the joint motion data are data generated when the real tester and the real robot arm move based on human-machine action labels;

[0018] The movement of the virtual digital human is driven based on the human body posture data, and the movement of the virtual robotic arm is driven based on the joint movement data, so that the movements of the virtual digital human and the real tester are synchronized, and the movements of the virtual robotic arm and the real robotic arm are synchronized, so that the virtual digital human and the virtual robotic arm can perform virtual human-machine collaboration in a virtual human-machine collaboration scene.

[0019] Optionally, the multi-angle human-machine scene image is an image obtained after video frame extraction and fixed-frequency acquisition of a shot video, wherein the shot video is a video shot by a group of virtual cameras set in a virtual human-machine collaboration scene, wherein the group of virtual cameras includes three virtual cameras, and the shooting angles of the three virtual cameras are orthogonal to each other;

[0020] The multi-angle human-machine scene image includes a human-machine scene image of a virtual human-machine collaboration scene from a frontal perspective, a human-machine scene image of a virtual human-machine collaboration scene from a side perspective, and a human-machine scene image of a virtual human-machine collaboration scene from a top perspective;

[0021] The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model also includes: using the human-machine action label as the name of the multi-angle human-machine scene image to name the multi-angle human-machine scene image.

[0022] Optionally, performing timestamp alignment on the human body collision part, the multi-angle human-machine scene image and the joint posture data specifically includes:

[0023] Determine the maximum and minimum values ​​of the timestamp of the human body collision part, the timestamp of the multi-angle human-machine scene image, and the timestamp of the joint posture data;

[0024] The time interval is determined based on the maximum value and the minimum value, and the timestamps of the human body collision part, the multi-angle human-machine scene image and the joint posture data are aligned by traversal and interpolation methods.

[0025] Optionally, converting the multimodal dataset into a visual-language instruction dataset using a multimodal macro model specifically includes:

[0026] Using the multimodal large model, automatically annotating the aligned multi-angle human-machine scene images in the multimodal dataset to obtain image titles;

[0027] Taking the multimodal dataset and the image caption as input, a visual-language instruction dataset is generated using a multimodal large model.

[0028] Optionally, the multimodal large model used to obtain the image title is CogVLM2, GLM4V, Qwen-VL-Chat or MiniCPM-V-2.5; the multimodal large model used to generate the visual-language instruction dataset is ChatGPT; the multimodal large model used to be fine-tuned to achieve anomaly detection is CogVLM2, GLM4V, Qwen-VL-Chat or MiniCPM-V-2.5.

[0029] Optionally, taking the multimodal dataset and the image caption as input, generating a visual-language instruction dataset using a multimodal large model specifically includes:

[0030] Constructing multiple dialogue templates; the dialogue templates are used to guide the multimodal large model to generate instruction data with specific patterns and styles, and the content of the dialogue templates includes the query image content and the reasoning scenario;

[0031] The multimodal dataset, the image caption and a plurality of dialogue templates are used as inputs, and a visual-language instruction dataset is generated using a multimodal large model; the visual-language instruction dataset includes multiple types of text data, and the types of text data include detailed description type data, long dialogue type data and complex reasoning type data.

[0032] Optionally, fine-tuning the multimodal large model using the visual-language instruction dataset to obtain a fine-tuned multimodal large model specifically includes:

[0033] Taking the visual-language instruction dataset as input, the multimodal large model is fine-tuned using a fine-tuning strategy based on training loss optimization to obtain a fine-tuned multimodal large model; wherein the fine-tuning strategy based on training loss optimization includes an AdamW optimizer, a cosine learning rate scheduler, and a mixed precision strategy.

[0034] Optionally, the fine-tuned multimodal large model is used to implement human-machine collaboration anomaly detection in a virtual human-machine collaboration scenario, specifically including:

[0035] Taking the currently acquired multi-angle human-machine scene image to be detected as input, the fine-tuned multimodal large model is used for anomaly detection to obtain the tester's body posture, the working state of the robotic arm, the pixel coordinates of the human body and the pixel coordinates of the robotic arm;

[0036] Performing coordinate transformation on the human body pixel coordinates to obtain human body space coordinates;

[0037] Performing coordinate transformation on the pixel coordinates of the robotic arm to obtain the spatial coordinates of the robotic arm;

[0038] Calculating the Euclidean distance between the human body space coordinates and the robotic arm space coordinates to obtain the human-machine space distance;

[0039] Determining a human-machine safety level based on the human-machine spatial distance and the minimum safe collaboration distance;

[0040] Wherein, determining the human-machine safety level based on the human-machine space distance and the minimum safe collaboration distance specifically includes:

[0041] When the human-machine space distance is less than or equal to the minimum safe cooperation distance, the human-machine safety level is dangerous braking;

[0042] When the human-machine space distance is greater than the minimum safe collaboration distance and less than or equal to the preset distance, the human-machine safety level is a warning reminder;

[0043] When the human-machine space distance is greater than the preset distance, the human-machine safety level is safe and normal;

[0044] The calculation method of the minimum safe collaboration distance includes:

[0045] Acquire multiple multi-angle human-machine scene images for sampling within a sampling time period; the sampling time period is (T0, T1+T), T0 is the time point when the human body breaks in, T1 is the time point when the robotic arm is triggered to retreat, and T is the time taken by the robotic arm from the beginning of retreat to the stop after retreating to a safe distance;

[0046] For each of the multi-angle human-machine scene images for sampling, the multi-angle human-machine scene images for sampling are used as input, and anomaly detection is performed using the fine-tuned multimodal large model to obtain pixel coordinates of a human body for sampling and pixel coordinates of a robotic arm for sampling;

[0047] For any two adjacent multi-angle human-machine scene images for sampling, the ratio of the sampling human body pixel coordinates of the two adjacent multi-angle human-machine scene images for sampling to the time interval of the two adjacent multi-angle human-machine scene images for sampling is calculated to obtain the moving speed of the sampling human body, and the ratio of the sampling mechanical arm pixel coordinates of the two adjacent multi-angle human-machine scene images for sampling to the time interval of the two adjacent multi-angle human-machine scene images for sampling is calculated to obtain the moving speed of the sampling mechanical arm;

[0048] Calculate the minimum safe collaboration distance according to the moving speed of the sampling human body and the moving speed of the sampling robotic arm;

[0049] The calculation formula for the minimum safe collaboration distance is:

[0050] Sp=Sh+Sr+Ss+C;

[0051] Among them, Sp is the minimum safe cooperation distance; Sh is the distance related to the human body approach speed, Vh(t) is the moving speed of the human body at time t, which is determined based on the human body moving speed used for sampling; Sr is the relevant distance of the normal working speed of the robot arm, Vr(t) is the moving speed of the robot arm at time t in the direction of approaching the human body, which is determined based on the moving speed of the robot arm for sampling; Ss is the stopping path distance of the robot arm, Vs(t) is the moving speed of the robot arm along the stopping path at time t, which is determined based on the moving speed of the robot arm used for sampling; C is the distance that the human body intrudes.

[0052] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0053] The present application provides a method for constructing a human-machine collaboration data set and detecting anomalies by integrating a multimodal large model. Based on human-machine action labels, a virtual digital human and a virtual robotic arm are controlled to perform virtual human-machine collaboration in a virtual human-machine collaboration scene. In the process of performing virtual human-machine collaboration, the human body collision part, multi-angle human-machine scene images of the virtual human-machine collaboration scene, and joint posture data of the virtual robotic arm are obtained when the virtual digital human collides with a physical asset in the virtual human-machine collaboration scene. The human body collision part, multi-angle human-machine scene images, and joint posture data are timestamped and aligned. The joint posture data constitutes the time series data of the human-machine collaboration virtual scene corresponding to the human-machine action label, and each human-machine action label and the human-machine collaboration virtual scene time series data corresponding to the human-machine action label constitute a multimodal data set. This application completes virtual human-machine collaboration and data collection by constructing a digital twin system, and does not require actual human-machine collaboration and data collection processes. It can solve the problems of time-consuming and labor-intensive debugging of complex hardware equipment, the need to manually label data, and the potential risk of collision. At the same time, it will collect multi-angle human-machine scene images to solve the problem that only single-angle images can be collected in actual scenes, thereby efficiently generating high-quality multimodal data sets. After obtaining the multimodal data set, the multimodal large model is further used to convert the multimodal data set into a visual-language instruction data set, and the multimodal large model is fine-tuned using the visual-language instruction data set to obtain the fine-tuned multimodal large model, and the fine-tuned multimodal large model is used to realize human-machine collaboration anomaly detection in virtual human-machine collaboration scenes. This application can efficiently construct a high-quality multimodal data set, and can fine-tune the multimodal large model to realize human-machine collaboration anomaly detection in virtual human-machine collaboration scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 This is an application environment diagram of a method for constructing a human-machine collaborative dataset and detecting anomalies that integrates a multimodal large model provided in Example 1 of the present application.

[0056] Figure 2 A flowchart of a method for constructing a human-machine collaborative data set and detecting anomalies that integrates a multimodal large model is provided in Example 1 of the present application.

[0057] Figure 3 A schematic diagram of the principles of a method for constructing a human-machine collaborative dataset and detecting anomalies that integrates a multimodal large model provided in Example 1 of the present application.

[0058] Figure 4 A schematic diagram of the human-machine action label categories provided in Example 1 of the present application.

[0059] Figure 5 This is a schematic diagram of a frontal perspective of a virtual human-machine collaboration scene captured at a certain moment provided in Example 1 of the present application.

[0060] Figure 6 A schematic diagram of a side view of a virtual human-machine collaboration scene captured at a certain moment provided in Example 1 of the present application.

[0061] Figure 7 A schematic diagram of a top-down perspective of a virtual human-machine collaboration scene captured at a certain moment provided in Example 1 of the present application.

[0062] Figure 8 A flowchart of the timestamp alignment algorithm provided in Example 1 of the present application.

[0063] Fig. 9 A schematic diagram of the subdivision types of the visual-language instruction dataset provided in Example 1 of the present application.

[0064] Fig.10 A schematic diagram of the structure of a computer device provided in Example 2 of the present application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] Example 1

[0067] The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model provided in the embodiments of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal communicates with the server through the network. The data storage system can store data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send a request for the construction of the dataset to be processed and anomaly detection (used to build a multimodal dataset and perform anomaly detection) to the server. After the server receives the request for the construction of the dataset to be processed and the anomaly detection request, for the request for the construction of the dataset to be processed and the anomaly detection request, the server controls the virtual digital human and the virtual robotic arm to perform virtual human-machine collaboration in the virtual human-machine collaboration scene based on the human-machine action label; in the process of virtual human-machine collaboration, obtain the human body collision part when the virtual digital human collides with the physical assets in the virtual human-machine collaboration scene, the multi-angle human-machine scene image of the virtual human-machine collaboration scene, and the joint posture data of the virtual robotic arm; for the human body collision part The human body collision parts, multi-angle human-machine scene images and joint posture data are timestamped, and the human body collision parts, multi-angle human-machine scene images and joint posture data after timestamp alignment are composed of human-machine collaborative virtual scene time series data corresponding to the human-machine action label; each human-machine action label and the human-machine collaborative virtual scene time series data corresponding to the human-machine action label are composed of a multimodal data set; the multimodal data set is converted into a visual-language instruction data set using a multimodal large model; the multimodal large model is fine-tuned using the visual-language instruction data set to obtain a fine-tuned multimodal large model; the fine-tuned multimodal large model is used to realize human-machine collaborative anomaly detection in virtual human-machine collaborative scenes. The server can feed back the obtained multimodal data set, the fine-tuned multimodal large model and the anomaly detection results to the terminal.

[0068] In addition, in some embodiments, the human-computer collaborative dataset construction and anomaly detection method for integrating a multimodal large model can also be implemented independently by a server or a terminal. For example, the terminal can directly process the dataset construction and anomaly detection requests to be processed, or the server can obtain the dataset construction and anomaly detection requests to be processed from the data storage system and process the dataset construction and anomaly detection requests to be processed.

[0069] The terminals may be, but are not limited to, various desktop computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers, or a cloud server.

[0070] like Figure 2 and Figure 3 As shown, this embodiment provides a method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0071] Step S1, controlling a virtual digital human and a virtual robotic arm to perform virtual human-machine collaboration in a virtual human-machine collaboration scene based on human-machine action labels.

[0072] Step S2, during the virtual human-machine collaboration process, obtain the human body collision part when the virtual digital human collides with the physical assets in the virtual human-machine collaboration scene, the multi-angle human-machine scene image of the virtual human-machine collaboration scene and the joint posture data of the virtual robotic arm.

[0073] Step S3, aligning the timestamps of the human body collision parts, the multi-angle human-machine scene images and the joint posture data, and composing the human body collision parts, multi-angle human-machine scene images and joint posture data after the timestamp alignment into the human-machine collaborative virtual scene timing data corresponding to the human-machine action label.

[0074] Step S4, forming a multimodal data set by combining each of the human-machine action labels and the human-machine collaborative virtual scene time series data corresponding to the human-machine action labels.

[0075] Step S5: using a multimodal large model to convert the multimodal dataset into a visual-language instruction dataset.

[0076] Step S6, using the visual-language instruction data set to fine-tune the multimodal large model to obtain a fine-tuned multimodal large model; using the fine-tuned multimodal large model to implement human-machine collaboration anomaly detection in a virtual human-machine collaboration scenario.

[0077] By implementing the above-mentioned steps S1 to S6, this embodiment provides a one-stop multimodal data collection and automatic label construction solution for human-computer collaboration scenarios, which solves the problems of long collection cycle and low efficiency in data labeling caused by cumbersome processes during actual machine debugging, avoids potential collision risks in human-computer collaboration scenarios, and can collect scene images from different angles, thereby efficiently building a high-quality multimodal data set, and can fine-tune the multimodal large model for training, and further use the fine-tuned multimodal large model to realize human-computer collaboration anomaly detection in virtual human-computer collaboration scenarios.

[0078] This embodiment uses a shooting strategy based on human-machine action labels to obtain human body collision parts, multi-angle human-machine scene images and joint posture data in the virtual human-machine collaboration process.

[0079] Among them, the core of the shooting strategy based on human-machine action labels is to use human-machine action labels to classify and mark scene information such as the tester's body posture, the working status of the robotic arm, the human-machine spatial distance, etc., as a target guide for human-machine collaborative behavior, to facilitate the subsequent automatic collection of multimodal data and automatic construction of labels in virtual human-machine collaborative scenes.

[0080] The categories of human-machine action labels are formulated based on the needs of specific human-machine collaborative tasks. Figure 4 As shown, in the human-machine action labels, the tester's human posture includes but is not limited to typical state labels such as standing, squatting (i.e. squatting and sitting down), and lying down. The working state of the robot arm includes but is not limited to key action stage labels such as initial stillness, movement, and grasping. Among them, movement includes translation and rotation of the robot arm, but the end of the robot arm does not grasp. Grasping refers to the end of the robot arm grasping the object. The human-machine space distance can be divided into a variety of labels that can reflect the degree of interaction safety according to the actual human-machine collaboration situation, including but not limited to safe and normal, warning reminder, dangerous braking and other interactive safety degree labels. Among them, when the human-machine space distance is less than or equal to the minimum safe collaboration distance, it corresponds to dangerous braking. When the human-machine space distance is greater than the minimum safe collaboration distance and less than or equal to the preset distance, it corresponds to warning reminder. When the human-machine space distance is greater than the preset distance, it corresponds to safe and normal. It should be noted that the preset distance is determined according to the safe operating distance of the specific human-machine collaboration task.

[0081] At this time, in this embodiment, the human-machine action labels include: a label of the tester's body posture, a label of the robotic arm's working status, and a label of the human-machine space distance. The labels of the tester's body posture include standing, squatting, sitting, and lying down. The labels of the robotic arm's working status include still, moving but not grasping, and grasping. The labels of the human-machine space distance include safe and normal, warning reminder, and dangerous braking.

[0082] This embodiment pre-builds a digital twin system for human-machine collaboration. Specifically, the real working environment of human-machine collaboration is modeled to obtain a virtual human-machine collaboration scene (also called a human-machine collaboration digital twin scene). Then, in the virtual human-machine collaboration scene, a virtual digital human (also called a digital twin human model) and a virtual robotic arm (also called a digital twin robotic arm model) are constructed that can sense in real time and perform virtual human-machine collaboration in the virtual human-machine collaboration scene. The virtual digital human is driven by a real tester and moves synchronously with the real tester. The virtual robotic arm is driven by a real robotic arm and moves synchronously with the real robotic arm.

[0083] The human posture of the real tester is mapped to the virtual digital human in real time through the virtual reality device, so that the virtual digital human and the real tester move synchronously. The virtual reality device includes a head-mounted display device worn on the head of the real tester, a handle worn on the hand of the real tester, and multiple positioning base stations installed in the test area where the real tester is located. The positioning base station is used to emit laser signals. The head-mounted display device and the handle are both equipped with a gyroscope and a laser receiver. The gyroscope is used to measure the posture of the real tester. The laser receiver is used to measure the position of the real tester based on the received laser signal. The head-mounted display device is used to watch the virtual human-machine collaboration between the virtual digital human and the virtual mechanical arm from a first-person perspective. The handle has buttons, which are used to control the movement and hand movements of the virtual digital human. The handle can control the virtual digital human to move forward, backward, left, and right, and can also control the virtual digital human to turn. It can also select and map hand movements for the virtual digital human. The hand movements can physically interact with the virtual human-machine collaboration scene, such as dragging the mechanical arm for teaching, grasping objects, and sliding doors. After obtaining the human posture data (i.e., the position and posture of the real tester) of the real tester through the virtual reality device, the virtual digital human is further driven to move based on the human posture data, specifically including: using the human posture data and the skeleton model of the virtual digital human as input, using the skeleton inverse kinematics solver to solve the joint data of the skeleton model, and driving the virtual digital human to move based on the joint data. When the handle has the function of selecting hand movements, the human posture data at this time also includes hand movements.

[0084] Among them, the joint motion data of the real robot arm is transmitted to the virtual robot arm in real time through the sensor, which further drives the movement of the virtual robot arm, so that the virtual robot arm is synchronized with the movement of the real robot arm. The sensor includes a speed sensor and an angle sensor installed at each joint of the real robot arm. Each joint of the virtual robot arm is set with a rotational pair constraint and a joint angle variable. After the joint motion data (i.e., the angle and speed of each joint) is obtained through the sensor, the virtual robot arm is driven to move based on the joint motion data, specifically including: using the joint motion data as input, using the state updater to update the joint angle variable of each joint of the virtual robot arm, and driving the virtual robot arm to move.

[0085] At this time, in this embodiment, controlling the virtual digital human and the virtual robotic arm to perform virtual human-machine collaboration in the virtual human-machine collaboration scene based on the human-machine action label specifically includes:

[0086] (1) Obtaining human posture data of a real test subject and joint motion data of a real robotic arm. The human posture data and joint motion data are data generated when the real test subject and the real robotic arm move based on human-machine action labels.

[0087] It should be noted that the real tester and the real robotic arm do not perform actual human-machine collaboration, but only complete their respective actions based on human-machine action labels, so that the virtual digital human and the virtual robotic arm can perform the same virtual human-machine collaboration as the actual human-machine collaboration.

[0088] (2) The movement of the virtual digital human is driven based on the human posture data, and the movement of the virtual robotic arm is driven based on the joint movement data, so that the movement of the virtual digital human is synchronized with that of the real tester, and the movement of the virtual robotic arm is synchronized with that of the real robotic arm, so that the virtual digital human and the virtual robotic arm can perform virtual human-machine collaboration in a virtual human-machine collaboration scenario.

[0089] In this embodiment, during the process of virtual human-machine collaboration, the human body collision part when the virtual digital human collides with the physical asset in the virtual human-machine collaboration scene, the multi-angle human-machine scene image of the virtual human-machine collaboration scene, and the joint posture data of the virtual robotic arm are obtained. Among them, the human body collision part is collected when the collision occurs, the multi-angle human-machine scene image is collected at a fixed first frequency, and the joint posture data of the virtual robotic arm is collected at a fixed second frequency.

[0090] The method of obtaining the human body collision part is: judging whether the virtual digital human and the physical assets in the virtual human-machine collaboration scene have a physical collision. If so, determining and recording the human body collision part and updating the human-machine state; if not, updating the human-machine state. Determining and recording the human body collision part specifically includes: using the collision event distributor to record the human body joint posture data of the virtual digital human in the previous frame of the collision and the human body joint posture data of the virtual digital human in the current frame of the collision, respectively, to obtain posture recording data; based on the posture recording data, using the ray collision detection function to determine the human body collision part; recording the human body collision part.

[0091] The method for acquiring multi-angle human-machine scene images is as follows: this embodiment builds a multi-angle image automatic acquisition system in a virtual human-machine collaboration scene, that is, the multi-angle image automatic acquisition system is built based on the virtual human-machine collaboration scene. Specifically, a group of virtual cameras are set at three orthogonal positions of the virtual human-machine collaboration scene to capture the dynamic changes of the virtual human-machine collaboration scene in real time from three different perspectives: front view, side view, and top view. Specifically, the three-dimensional coordinate system in the Unreal Engine is used, and the normal vector of the xoz plane corresponds to the front view, the normal vector of the yoz plane corresponds to the side view, and the normal vector of the xoy plane corresponds to the top view, to obtain three captured videos. For each captured video, it is converted into a frame-by-frame image through the scene texture capture component, and the frame-by-frame image is further acquired at a fixed first frequency to obtain a multi-angle human-machine scene image, and the multi-angle human-machine scene image is saved in a local folder. The human-machine action label corresponding to the multi-angle human-machine scene image is used as the name of the multi-angle human-machine scene image through a program script. Among them, the scene texture capture component is a functional component in the software (such as Unreal Engine) used to create a digital twin system, and the program script is a user-defined program whose function is to use the human-machine action label corresponding to the multi-angle human-machine scene image as the name of the multi-angle human-machine scene image.

[0092] At this time, the multi-angle human-machine scene image is an image obtained by extracting video frames and collecting fixed-frequency data from a shot video. The shot video is shot by a group of virtual cameras set in a virtual human-machine collaboration scene. The group of virtual cameras includes three virtual cameras, and the shooting angles of the three virtual cameras are mutually orthogonal. Specifically, the multi-angle human-machine scene image includes a human-machine scene image of the virtual human-machine collaboration scene from a front view angle, a human-machine scene image of the virtual human-machine collaboration scene from a side view angle, and a human-machine scene image of the virtual human-machine collaboration scene from a top view angle, such as Figure 5 , Figure 6 and Figure 7 shown.

[0093] After acquiring multi-angle human-machine scene images, the method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model in this embodiment further includes: using human-machine action labels as names of multi-angle human-machine scene images to name the multi-angle human-machine scene images.

[0094] The joint posture data is acquired by collecting the joint postures of the virtual robotic arm at a fixed second frequency to obtain the joint posture data, wherein the joint posture data includes the angle and speed of each joint in the virtual robotic arm.

[0095] This embodiment combines the human body collision parts of the virtual digital human, the multi-angle human-machine scene images of the virtual human-machine collaboration scene, and the joint posture data of the virtual robot arm, and constructs the human-machine collaboration virtual scene time series data based on human-machine action labels through the timestamp alignment algorithm. Figure 8 As shown in the figure, first, the timestamps of the human body collision parts, the timestamps of the multi-angle human-machine scene images, and the timestamps of the joint posture data are extracted. Then, the minimum and maximum values ​​of all timestamps are determined to define the time range and determine the time interval. Then, the timestamp data from different sources (i.e., the human body collision parts, multi-angle human-machine scene images, and joint posture data) are aligned to a unified time interval through traversal and interpolation methods. The human body collision parts, multi-angle human-machine scene images, and joint posture data are integrated according to the aligned timestamp sequence to achieve time series alignment and obtain the human-machine collaborative virtual scene time series data. Finally, the three types of time series information of the aligned human body collision parts, multi-angle human-machine scene images, and joint posture data are fused with the human-machine action labels, so that the human-machine collaborative virtual scene time series data is integrated into the classification labels of different human-machine collaborative behaviors, forming multimodal original data with spatiotemporal characteristics, reflecting the dynamic changes in the human-machine collaborative scene.

[0096] At this time, in this embodiment, the human body collision parts, multi-angle human-machine scene images and joint posture data are timestamp aligned, and the human body collision parts, multi-angle human-machine scene images and joint posture data after timestamp alignment are composed of human-machine collaborative virtual scene time series data corresponding to the human-machine action label, wherein the human body collision parts, multi-angle human-machine scene images and joint posture data are timestamp aligned, specifically including: determining the maximum and minimum values ​​of the timestamps of the human body collision parts, the timestamps of the multi-angle human-machine scene images and the timestamps of the joint posture data; determining the time interval based on the maximum and minimum values, and timestamp aligning the human body collision parts, multi-angle human-machine scene images and joint posture data through traversal and interpolation methods.

[0097] After obtaining the time series data of the human-machine collaboration virtual scene, this embodiment further combines each human-machine action label and the time series data of the human-machine collaboration virtual scene corresponding to the human-machine action label into a multimodal data set, thereby completing the construction of a multimodal data set based on the virtual human-machine collaboration scene.

[0098] This embodiment uses a multimodal large model to convert a multimodal dataset of a virtual human-machine collaboration scenario into a visual-language instruction dataset in a visual-language instruction format, and further improves the adaptability of the multimodal large model in the human-machine collaboration scenario through supervised fine-tuning training.

[0099] A multimodal large model is used to convert a multimodal dataset of a virtual human-machine collaboration scene into a visual-language instruction dataset in a visual-language instruction format, specifically including: using multimodal large models such as CogVLM2, GLM4V, Qwen-VL-Chat or MiniCPM-V-2.5 to automatically annotate virtual scene images (i.e., aligned multi-angle human-machine scene images) in the multimodal dataset to generate image titles. Image titles refer to text descriptions generated for the visual information contained in the multi-angle human-machine scene images. The text descriptions include specific visual information, such as human body postures, human body collision sites, and other situation-related elements. Their function is to help establish the association between vision and language, so that the multimodal large model can understand the content in the multi-angle human-machine scene images and convert them into text descriptions. The image title describes the key visual elements in the multi-angle human-machine scene images. For example, the image title is: A person in a standing position is interacting with a robotic arm, and the left limb collides. Pre-building 100 dialogue templates as seed samples can help guide multimodal large models to generate instruction data with specific patterns and styles. The content of the dialogue templates can involve possible ways of interaction between users and assistants, such as asking about image content, reasoning scenarios, etc. For example, the dialogue templates are: "Please describe the human posture in the image.", "Are there any potentially dangerous conditions in this scene?", "How do the movements of the robot arm interact with the position of the human body?". Combined with multimodal datasets, image captions, and seed samples, multimodal large models such as ChatGPT are guided to generate large-scale, high-quality visual-language instruction datasets for instruction tuning, such as Fig. 9 As shown, the visual-language instruction dataset includes three types of text data:

[0100] (1) Detailed description data: Each sample contains a round of dialogue, which mainly describes the detailed information of the image and its label. Detailed description data is a single round of description, emphasizing the comprehensiveness of the image content and constructing a rich and comprehensive explanation. For example, detailed description data can be:

[0101]

[0102]

[0103] (2) Long dialogue data: Each sample involves multiple rounds of dialogue, simulating the process of interaction between the user and the assistant on the image content. Long dialogue data is used to simulate multiple rounds of dialogue and gradually gain a deeper understanding of anomaly detection in virtual human-computer collaboration, such as asking about the safe distance of human-computer interaction, abnormal status, etc. For example, long dialogue data can be:

[0104]

[0105]

[0106] (3) Complex reasoning data: Each sample focuses on scenarios and problems that require visual reasoning. Complex reasoning data is used to promote the model to generate visual-based reasoning analysis, such as inferring abnormal situations based on image content, detecting possible safety risks, etc. For example, complex reasoning data can be:

[0107]

[0108] At this time, in this embodiment, the multimodal data set is converted into a visual-language instruction data set using a multimodal large model, specifically including:

[0109] (1) Use the multimodal large model to automatically annotate the aligned multi-angle human-machine scene images in the multimodal dataset to obtain image titles.

[0110] Among them, the multimodal large model used to obtain image titles is CogVLM2, GLM4V, Qwen-VL-Chat or MiniCPM-V-2.5.

[0111] (2) Taking a multimodal dataset and image captions as input, a multimodal large model is used to generate a visual-language instruction dataset.

[0112] Among them, taking the multimodal dataset and image title as input, using the multimodal large model to generate a visual-language instruction dataset, specifically including: constructing multiple dialogue templates, the dialogue templates are used to guide the multimodal large model to generate instruction data with specific patterns and styles, and the content of the dialogue templates includes asking about image content and reasoning scenarios; taking the multimodal dataset, image title and multiple dialogue templates as input, using the multimodal large model to generate a visual-language instruction dataset, the visual-language instruction dataset includes multiple types of text data, and the types of text data include detailed description type data, long dialogue type data and complex reasoning type data.

[0113] Among them, the multimodal large model used to generate visual-language instruction datasets is ChatGPT.

[0114] It should be noted that each sample in the visual-language instruction dataset still includes the aligned human body collision parts, multi-angle human-machine scene images and joint posture data, as well as human-machine action labels, but these data information are described in the form of natural language.

[0115] The adaptability of the multimodal large model in human-machine collaboration scenarios is improved through supervised fine-tuning training, specifically including: fine-tuning the multimodal large model through the constructed visual-language instruction dataset. During the fine-tuning process, a fine-tuning strategy based on training loss optimization is adopted. The fine-tuning strategy based on training loss optimization adopts the AdamW optimizer, cosine learning rate scheduler and a mixed precision strategy combining float16 and float32. Subsequently, the fine-tuned multimodal large model is used to identify abnormal states in the human-machine collaboration process, realize anomaly detection, and further respond to abnormal states to generate text responses.

[0116] At this time, in this embodiment, the multimodal large model is fine-tuned using the visual-language instruction dataset to obtain a fine-tuned multimodal large model, specifically including: taking the visual-language instruction dataset as input, and fine-tuning the multimodal large model using a fine-tuning strategy based on training loss optimization to obtain a fine-tuned multimodal large model, wherein the fine-tuning strategy based on training loss optimization includes an AdamW optimizer, a cosine learning rate scheduler, and a mixed precision strategy.

[0117] Among them, the multimodal large model used to be fine-tuned to achieve anomaly detection is CogVLM2, GLM4V, Qwen-VL-Chat or MiniCPM-V-2.5.

[0118] The fine-tuned multimodal large model is used to realize human-machine collaboration anomaly detection in a virtual human-machine collaboration scene. The fine-tuned multimodal large model is used to realize human-machine collaboration anomaly detection in a virtual human-machine collaboration scene, specifically including: taking the currently acquired multi-angle human-machine scene image to be detected as input, using the fine-tuned multimodal large model to perform anomaly detection, and obtaining the tester's human posture, robotic arm working state, human body pixel coordinates and robotic arm pixel coordinates; coordinate transformation of the human body pixel coordinates to obtain human body space coordinates; coordinate transformation of the robotic arm pixel coordinates to obtain robotic arm space coordinates; calculating the Euclidean distance between the human body space coordinates and the robotic arm space coordinates to obtain the human-machine space distance; determining the human-machine safety level based on the human-machine space distance and the minimum safe collaboration distance, specifically when the human-machine space distance is less than or equal to the minimum safe collaboration distance, the human-machine safety level is dangerous braking, when the human-machine space distance is greater than the minimum safe collaboration distance and less than or equal to the preset distance, the human-machine safety level is warning reminder, when the human-machine space distance is greater than the preset distance, the human-machine safety level is safe and normal, specifically including the following steps:

[0119] (1) Coordinate extraction and analysis: The fine-tuned multi-modal large model first receives the input multi-angle human-machine scene image to be detected. Based on its own multi-modal capabilities, the fine-tuned multi-modal large model further combines the vision pre-training model (ViT-biG) of the target task to analyze the content of the multi-angle human-machine scene image to be detected, and generates coordinate frames and key points of the human body and the robotic arm (i.e., human pixel coordinates and robotic arm pixel coordinates).

[0120] (2) Calculation of human-machine coordinates and spatial relationship: Through the detected coordinate frames and key points of the human body and the robotic arm, calculate the human body posture of the tester, the working state of the robotic arm, and the human-machine spatial distance to judge the relative spatial position of the human body and the robotic arm. The calculation method of the human-machine spatial distance is as follows: Based on the obtained human pixel coordinates and robotic arm pixel coordinates, use the spatial distance calculation module to calculate the human-machine spatial distance. The spatial distance calculation module is responsible for extracting human pixel coordinates and robotic arm pixel coordinates from the image, converting the detected pixel coordinates into actual spatial coordinates (x, y, z), specifically using a multi-angle camera for triangulation to convert the extracted 2D pixel coordinates into 3D spatial coordinates, obtaining the human body spatial coordinates and the robotic arm spatial coordinates. By calculating the Euclidean distance between two sets of coordinate points (i.e., human body spatial coordinates and robotic arm spatial coordinates), the human-machine spatial distance is obtained: where D is the human-machine spatial distance, (x1, y1, z1) represents the human body spatial coordinates, and (x2, y2, z2) represents the robotic arm spatial coordinates.

[0121] (3) Application of abnormal rules: After extracting the human-machine spatial distance, based on the rule library of collision and safety, this rule library can be automatically learned and optimized by a pre-defined training data set during the fine-tuning process. Specifically, the spatial distance calculation module judges the human-machine safety level according to the calculated human-machine spatial distance D: If D ≤ the minimum safe cooperation distance, it is judged as dangerous braking; if the minimum safe cooperation distance < D ≤ D1, D1 is a preset distance, it is judged as a warning reminder; if D > D1, it is judged as safe and normal, and the final human-machine safety level is fed back to the multi-modal large model.

[0122] (4) Abnormal type discrimination and response generation: When an abnormal situation (such as too close distance or collision) is detected, the model can generate a corresponding text response.

[0123] Among them, the calculation method of the minimum safe cooperation distance includes:

[0124] (1) Acquire multiple multi-angle human-machine scene images for sampling within a sampling time period, where the sampling time period is (T0, T1+T), where T0 is the time point when the human body breaks into the virtual human-machine collaboration scene, that is, the specific time point when the human body is detected to have just entered the virtual human-machine collaboration scene, T1 is the time point when the distance between the human body and the robotic arm is less than a certain distance, triggering the robotic arm to retreat, and T is the time taken by the robotic arm to retreat from the beginning to the end after retreating to a safe distance and stopping, which is a preset value determined by the manufacturer.

[0125] (2) For each multi-angle human-machine scene image for sampling, the multi-angle human-machine scene image for sampling is used as input, and anomaly detection is performed using the fine-tuned multimodal large model to obtain the pixel coordinates of the sampling human body and the pixel coordinates of the sampling robotic arm.

[0126] (3) For any two adjacent multi-angle human-machine scene images for sampling, calculate the ratio of the sampling human body pixel coordinates of the two adjacent multi-angle human-machine scene images for sampling to the time interval between the two adjacent multi-angle human-machine scene images for sampling, and obtain the sampling human body movement speed; calculate the ratio of the sampling robot arm pixel coordinates of the two adjacent multi-angle human-machine scene images for sampling to the time interval between the two adjacent multi-angle human-machine scene images for sampling, and obtain the sampling robot arm movement speed.

[0127] Sample two adjacent input images and calculate the moving speeds of the human body and the robotic arm within the time interval dt, where the moving speed of the human body is obtained by dividing the change of the human body pixel coordinates in the two adjacent images by dt, and the moving speed of the robotic arm is obtained by dividing the change of the robotic arm pixel coordinates in the two adjacent images by dt.

[0128] (4) Calculate the minimum safe collaboration distance based on the moving speed of the sampling human body and the moving speed of the sampling robotic arm.

[0129] The calculation formula for the minimum safe collaboration distance is:

[0130] Sp=Sh+Sr+Ss+C;

[0131] Among them, Sp is the minimum safe cooperation distance; Sh is the distance related to the human body approach speed, Vh(t) is the moving speed of the human body at time t, which is determined based on the human body moving speed used for sampling; Sr is the relevant distance of the normal working speed of the robot arm, Vr(t) is the moving speed of the robot arm at time t in the direction of approaching the human body, which is determined based on the moving speed of the robot arm for sampling; Ss is the stopping path distance of the robot arm, Vs(t) is the moving speed of the robot arm along the stopping path at time t, which is determined based on the moving speed of the sampled robot arm and is generally determined by the manufacturer; C is the distance of human intrusion, that is, the distance between the human body and a specific boundary (the human body enters the virtual human-machine collaboration scene through this specific boundary) when the human body is detected as just entering the virtual human-machine collaboration scene.

[0132] Generating text responses means that when an abnormal situation is detected, the model generates appropriate text responses based on the learned visual-language instruction dataset to prompt the user to take measures or make adjustments. That is, text responses refer to the text that is automatically generated based on the detection results when the fine-tuned multimodal large model performs human-machine collaborative anomaly detection, which is used to remind users or provide solutions. Its content may include the following aspects:

[0133] (1) Description of abnormal status: The model can provide a brief description of the abnormal situation, for example, the robot arm is too close and there is a risk of collision.

[0134] (2) Safety advice or warnings: If a potential hazard is detected, the model may give a warning prompt, for example, please adjust the position of the robot arm to maintain a safe distance, or for example, the robot arm is currently in the wrong working mode.

[0135] (3) Solutions or action suggestions: If the abnormal state is related to a specific operation, the model may suggest how the user can adjust the working parameters or recalibrate the equipment, for example, placing the robot arm in its initial position and restarting.

[0136] (4) Task status update: In some cases, the model can also provide the status of the current collaborative task, for example, the task is paused and waiting for safety confirmation, or the task is continuing and the current status is safe.

[0137] As an example, suppose the model detects that the robotic arm is in a close posture to the human body and the distance is too close, and it is in a high-risk grasping stage. The model's text response may be: Warning: The current distance between the robotic arm and the human body is too close. There may be a risk of collision. Please adjust the position of the robotic arm to maintain a safe distance; Tip: Please make sure that the end of the robotic arm does not perform grasping operations to prevent accidents.

[0138] This embodiment proposes a method for constructing a human-machine collaboration dataset and detecting anomalies that integrates a multimodal large model, which relates to the field of human-machine collaboration technology. The method includes: building a multi-angle image automatic acquisition system in a virtual human-machine collaboration scene, using a shooting strategy based on human-machine action labels to obtain multi-angle human-machine scene images in the human-machine collaboration process, combining the human body collision parts, the joint posture data of the collaborative robot arm and the multi-angle human-machine scene images, and constructing human-machine collaboration virtual scene time series data based on human-machine action labels through a timestamp alignment algorithm, further constructing a multimodal dataset, using a multimodal large model to convert the multimodal dataset into a visual-language instruction dataset, and improving the adaptability of the multimodal large model in the human-machine collaboration scene through supervised fine-tuning training, and subsequently using the fine-tuned multimodal large model to achieve anomaly detection. This embodiment provides a one-stop human-machine collaboration scene multimodal data acquisition and automatic label construction solution to meet the current training needs of multimodal large models and improve the scene generalization understanding and intelligent detection level of human-machine collaboration tasks.

[0139] The method of this embodiment has the following advantages:

[0140] (1) A one-stop multimodal data collection and automatic labeling solution for human-machine collaboration scenarios solves the problems of long collection cycles and low efficiency in data labeling caused by cumbersome processes during actual machine debugging, avoids potential collision risks in human-machine collaboration scenarios, and can collect scene images from different angles.

[0141] (2) The time series data of the virtual scene of human-machine collaboration integrates different modal data such as the collision parts of the human body, the joint posture data of the collaborative robot arm, and multi-angle human-machine scene images, providing more dimensional and detailed scene information data for human-machine collaboration.

[0142] (3) A large-scale, high-quality visual-language instruction dataset that meets the current training requirements for multimodal large-model tasks and improves the scene generalization understanding and intelligent detection level of human-machine collaborative tasks.

[0143] The embodiment of the present application also provides an application scenario, which applies the above-mentioned human-machine collaborative dataset construction and anomaly detection method for the fused multimodal large model. Specifically, the human-machine collaborative dataset construction and anomaly detection method for the fused multimodal large model provided in this embodiment can be applied in the human-machine collaborative anomaly detection scenario. The human-machine collaborative anomaly detection scenario includes a dataset construction link, a model fine-tuning link, and anomaly detection link. The dataset construction link is used to construct a multimodal dataset based on a digital twin method, and the model fine-tuning link is used to use the multimodal dataset to fine-tune the multimodal large model to obtain the fine-tuned multimodal large model. The anomaly detection link is used to use the fine-tuned multimodal large model to perform anomaly detection in the human-machine collaborative process. The human-machine collaborative dataset construction and anomaly detection method for the fused multimodal large model provided in this embodiment belongs to the dataset construction link, the model fine-tuning link, and the anomaly detection link.

[0144] Example 2

[0145] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for constructing a human-computer collaborative data set and detecting anomalies that integrates a multi-modal large model is implemented.

[0146] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0147] In an exemplary embodiment, a computer device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for constructing a human-computer collaborative dataset and detecting anomaly of a fused multimodal large model as described in Example 1.

[0148] Example 3

[0149] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for constructing a human-computer collaborative data set and detecting anomalies for a large fusion multimodal model described in Example 1 is implemented.

[0150] Example 4

[0151] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method for constructing a human-machine collaborative data set and detecting anomalies for a large multimodal fusion model as described in Example 1.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0153] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for constructing a human-machine collaborative dataset and detecting anomalies by integrating a multimodal large model, characterized in that: The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model includes: Based on human-machine action labels, the virtual digital human and virtual robotic arm are controlled to perform virtual human-machine collaboration in a virtual human-machine collaboration scenario; In the process of virtual human-machine collaboration, the human body collision part when the virtual digital human collides with the physical assets in the virtual human-machine collaboration scene, the multi-angle human-machine scene images of the virtual human-machine collaboration scene and the joint posture data of the virtual robotic arm are obtained; Performing timestamp alignment on the human body collision part, the multi-angle human-machine scene image and the joint posture data, and composing the human body collision part, the multi-angle human-machine scene image and the joint posture data after the timestamp alignment into human-machine collaborative virtual scene time series data corresponding to the human-machine action label; Combining each of the human-machine action labels and the human-machine collaborative virtual scene time series data corresponding to the human-machine action labels into a multimodal data set; Converting the multimodal dataset into a visual-language instruction dataset using a multimodal macro model; The multimodal large model is fine-tuned using the visual-language instruction data set to obtain a fine-tuned multimodal large model; and the fine-tuned multimodal large model is used to implement human-machine collaboration anomaly detection in a virtual human-machine collaboration scenario.

2. The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model according to claim 1 is characterized in that: The human-machine action labels include: labels of the tester's body posture, labels of the robot arm's working status, and labels of the human-machine space distance; the labels of the tester's body posture include standing, squatting, sitting, and lying down; the labels of the robot arm's working status include still, moving but not grasping, and grasping; the labels of the human-machine space distance include safe and normal, warning reminder, and dangerous braking.

3. The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model according to claim 1 is characterized in that: Based on human-machine action tags, the virtual digital human and the virtual robotic arm are controlled to perform virtual human-machine collaboration in the virtual human-machine collaboration scene, including: Acquire human body posture data of a real tester and joint motion data of a real robot arm; the human body posture data and the joint motion data are data generated when the real tester and the real robot arm move based on human-machine action labels; The movement of the virtual digital human is driven based on the human body posture data, and the movement of the virtual robotic arm is driven based on the joint movement data, so that the movements of the virtual digital human and the real tester are synchronized, and the movements of the virtual robotic arm and the real robotic arm are synchronized, so that the virtual digital human and the virtual robotic arm can perform virtual human-machine collaboration in a virtual human-machine collaboration scene.

4. The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model according to claim 1 is characterized in that: The multi-angle human-machine scene image is an image obtained by extracting video frames and collecting them at a fixed frequency from a shot video, wherein the shot video is a video shot by a group of virtual cameras set in a virtual human-machine collaboration scene, wherein the group of virtual cameras includes three virtual cameras, and the shooting angles of the three virtual cameras are orthogonal to each other; The multi-angle human-machine scene image includes a human-machine scene image of a virtual human-machine collaboration scene from a frontal perspective, a human-machine scene image of a virtual human-machine collaboration scene from a side perspective, and a human-machine scene image of a virtual human-machine collaboration scene from a top perspective; The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model also includes: using the human-machine action label as the name of the multi-angle human-machine scene image to name the multi-angle human-machine scene image.

5. The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model according to claim 1 is characterized in that: Performing timestamp alignment on the human body collision part, the multi-angle human-machine scene image and the joint posture data specifically includes: Determine the maximum and minimum values ​​of the timestamp of the human body collision part, the timestamp of the multi-angle human-machine scene image, and the timestamp of the joint posture data; The time interval is determined based on the maximum value and the minimum value, and the timestamps of the human body collision part, the multi-angle human-machine scene image and the joint posture data are aligned by traversal and interpolation methods.

6. The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model according to claim 1 is characterized in that: The multimodal dataset is converted into a visual-language instruction dataset using a multimodal macro model, specifically comprising: Using the multimodal large model, automatically annotating the aligned multi-angle human-machine scene images in the multimodal dataset to obtain image titles; Taking the multimodal dataset and the image caption as input, a visual-language instruction dataset is generated using a multimodal large model.

7. The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model according to claim 6 is characterized in that: The multimodal large model used to obtain image titles is CogVLM2, GLM4V, Qwen-VL-Chat or MiniCPM-V-2.5; the multimodal large model used to generate visual-language instruction datasets is ChatGPT; the multimodal large model used to be fine-tuned for anomaly detection is CogVLM2, GLM4V, Qwen-VL-Chat or MiniCPM-V-2.

5.

8. The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model according to claim 7 is characterized in that: Taking the multimodal dataset and the image caption as input, generating a visual-language instruction dataset using a multimodal large model, specifically comprising: Constructing multiple dialogue templates; the dialogue templates are used to guide the multimodal large model to generate instruction data with specific patterns and styles, and the content of the dialogue templates includes the query image content and the reasoning scenario; The multimodal dataset, the image caption and a plurality of dialogue templates are used as inputs, and a visual-language instruction dataset is generated using a multimodal large model; the visual-language instruction dataset includes multiple types of text data, and the types of text data include detailed description type data, long dialogue type data and complex reasoning type data.

9. The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model according to claim 1, characterized in that: The multimodal large model is fine-tuned using the visual-language instruction dataset to obtain a fine-tuned multimodal large model, specifically comprising: Taking the visual-language instruction dataset as input, the multimodal large model is fine-tuned using a fine-tuning strategy based on training loss optimization to obtain a fine-tuned multimodal large model; wherein the fine-tuning strategy based on training loss optimization includes an AdamW optimizer, a cosine learning rate scheduler, and a mixed precision strategy.

10. The method for constructing a human-machine collaborative data set and detecting anomalies by integrating a multimodal large model according to claim 1, characterized in that: The fine-tuned multimodal large model is used to realize human-machine collaboration anomaly detection in virtual human-machine collaboration scenarios, including: Taking the currently acquired multi-angle human-machine scene image to be detected as input, the fine-tuned multimodal large model is used for anomaly detection to obtain the tester's body posture, the working state of the robotic arm, the pixel coordinates of the human body and the pixel coordinates of the robotic arm; Performing coordinate transformation on the human body pixel coordinates to obtain human body space coordinates; Performing coordinate transformation on the pixel coordinates of the robotic arm to obtain the spatial coordinates of the robotic arm; Calculating the Euclidean distance between the human body space coordinates and the robotic arm space coordinates to obtain the human-machine space distance; Determining a human-machine safety level based on the human-machine spatial distance and the minimum safe collaboration distance; Wherein, determining the human-machine safety level based on the human-machine space distance and the minimum safe collaboration distance specifically includes: When the human-machine space distance is less than or equal to the minimum safe cooperation distance, the human-machine safety level is dangerous braking; When the human-machine space distance is greater than the minimum safe collaboration distance and less than or equal to the preset distance, the human-machine safety level is a warning reminder; When the human-machine space distance is greater than the preset distance, the human-machine safety level is safe and normal; The calculation method of the minimum safe collaboration distance includes: Acquire multiple multi-angle human-machine scene images for sampling within a sampling time period; the sampling time period is (T0, T1+T), T0 is the time point when the human body breaks in, T1 is the time point when the robotic arm is triggered to retreat, and T is the time taken by the robotic arm from the beginning of retreat to the stop after retreating to a safe distance; For each of the multi-angle human-machine scene images for sampling, the multi-angle human-machine scene images for sampling are used as input, and anomaly detection is performed using the fine-tuned multimodal large model to obtain pixel coordinates of a human body for sampling and pixel coordinates of a robotic arm for sampling; For any two adjacent multi-angle human-machine scene images for sampling, the ratio of the sampling human body pixel coordinates of the two adjacent multi-angle human-machine scene images for sampling to the time interval of the two adjacent multi-angle human-machine scene images for sampling is calculated to obtain the moving speed of the sampling human body, and the ratio of the sampling mechanical arm pixel coordinates of the two adjacent multi-angle human-machine scene images for sampling to the time interval of the two adjacent multi-angle human-machine scene images for sampling is calculated to obtain the moving speed of the sampling mechanical arm; Calculate the minimum safe collaboration distance according to the moving speed of the sampling human body and the moving speed of the sampling robotic arm; The calculation formula for the minimum safe collaboration distance is: Sp=Sh+Sr+Ss+C; Among them, Sp is the minimum safe cooperation distance; Sh is the distance related to the human body approach speed, Vh(t) is the moving speed of the human body at time t, which is determined based on the human body moving speed used for sampling; Sr is the relevant distance of the normal working speed of the robot arm, Vr(t) is the moving speed of the robot arm at time t in the direction of approaching the human body, which is determined based on the moving speed of the robot arm for sampling; Ss is the stopping path distance of the robot arm, Vs(t) is the moving speed of the robot arm along the stopping path at time t, which is determined based on the moving speed of the robot arm used for sampling; C is the distance that the human body intrudes.

Images