Human body posture multi-view visual identification AI training data set automatic generation and identification method based on simulation environment

Abstract

A human body posture multi-view visual identification AI training data set automatic generation and identification method based on a simulation environment is characterized in that firstly, camera parameter matching is carried out; accurately configuring the cameras in the digital twin system according to the key parameters of the actual visual inspection camera; 2, industrial scene modeling and illumination simulation; performing accurate modeling and scene construction according to an actual industrial scene in a digital twin system; thirdly, performing digital figure customization and motion simulation; fourthly, training data are automatically output and identified; and outputting annotation files and pictures required by machine vision training through a twin system, wherein the annotation files comprise identity feature description of a character, joint data, a shielding relationship of joints, a character range frame, scene features, a screen range and information required by an internal and external parameter matrix of a camera. According to the method, a large number of high-quality and high-efficiency data sets are provided for human body posture machine vision AI training in an industrial scene. And the cost is reduced during data acquisition.

Description

Technical Field

[0001] The present invention relates to a method for automatically generating and identifying a human posture multi-view visual recognition AI training data set based on a simulation environment, and is particularly directed to the automatic generation and identification of human postures and scenes in an industrial environment. The method can be used in a digital twin environment to solve the problem of lack of human posture simulation data sets in industrial scenes. Background Art

[0002] With the rapid development of information technology, digital twin technology has become a hot topic in academic and industrial research. Digital twins use digital means to establish a dynamic virtual model of physical entities with multiple dimensions, multiple time and space scales, multiple disciplines, and multiple physical quantities to simulate and characterize the properties, behaviors, and rules of physical entities in real environments.

[0003] In this context, "virtual digital human", as an important branch of digital twin technology, has attracted more and more attention from the industry. The so-called "virtual digital human", or "digital human" for short, refers to the digital avatar of a real-world physical person generated in the virtual world using a variety of scientific and technological means. At present, digital human technology has been widely used in many fields such as games, entertainment, and sports.

[0004] In the field of machine vision, the training of human posture recognition AI requires a large amount of multi-view, multi-scene human motion data and key point annotations. In recent years, with the advancement of technology, the training of machine vision recognition of human posture has made significant progress, and the required data sets have become more complete and large. However, due to the particularity of the industrial field and the complexity of human movements in industrial environments, the content of traditional data sets in this field is seriously insufficient, resulting in limited scalability and restricting the development of visual recognition training, which is specifically manifested in the following aspects:

[0005] First, human postures in industrial environments are more complex and diverse, human movements are more likely to reach extreme states, and are more professional, which are significantly different from human movements in daily life.

[0006] Secondly, the production working environment in the industrial field is relatively complex, and obtaining relevant data is time-consuming and labor-intensive, and some of the data also have confidentiality issues.

[0007] Finally, the labeling information of key points of the human body needs to be more accurate, but the current data set key point labeling is mostly manual, with large errors and low efficiency, which can hardly meet the requirements of the industrial field.

[0008] Existing datasets are mainly derived from living environments. The scenes, lighting, human movements and other contents are quite different from those in industrial environments. If they are used for visual recognition training in the industrial field, serious deviations will occur. In addition, the accuracy and speed of annotation information in existing datasets cannot meet the requirements of high accuracy and high efficiency in industrial environments. Therefore, it is particularly urgent to establish a human posture dataset suitable for industrial scenarios.

[0009] In the industrial field, due to its particularity, it is often challenging to obtain data sets about human posture, lighting conditions and environmental factors. However, the application of digital twin technology provides a solution to this problem. Digital twin technology can simulate simulation effects close to the real world, allowing elements such as human models, movements, environments, lighting and camera angles to be freely combined. Studies have shown that the data sets generated in this way are comparable to actual data training sets in terms of authenticity and diversity. In addition, the use of these data sets for machine vision recognition training can also achieve the same level as real data training sets.

[0010] At the same time, the industrial field has more stringent requirements for the accuracy and efficiency of key point annotation. The multi-person key point detection algorithms used in current data sets are mostly top-down key point detection methods. This method first determines the approximate position of a single person through a target detection algorithm, and then performs key point detection on a single person within the image detection frame, thereby converting the multi-person key point detection problem into a single person key point detection problem. This method performs well in terms of accuracy, but its processing speed is relatively slow. Although it can meet the needs of the current data set, it still has a significant gap compared to the high standards of speed and accuracy in the industrial field. Therefore, the industrial field needs more efficient and accurate algorithms to improve the performance of key point detection to meet its dual requirements for high efficiency and high accuracy.

[0011] In view of the environmental particularity of industrial scenes, the complex and professional nature of human movements, and the accuracy and efficiency requirements of key point annotation information of the human body, there is a relative lack of machine vision recognition training data in the industrial field, which limits the in-depth development of visual recognition training in this field. Therefore, a simulation environment automatic generation and identification method for human posture machine vision recognition AI training data is proposed. This method is suitable for the digital twin environment and aims to solve the problem of insufficient human posture simulation data sets in industrial scenes. Summary of the invention

[0012] The purpose of the present invention is to solve the problem of lack of AI training data sets for machine vision recognition of human posture in industrial environments, and to invent a method for automatically generating and identifying AI training data sets for multi-view vision recognition of human posture based on a simulation environment.

[0013] The technical solution of the present invention is:

[0014] A method for automatically generating and identifying AI training data sets for multi-view visual recognition of human posture based on a simulation environment, characterized by:

[0015] First, camera parameter matching is performed. The camera in the digital twin system is precisely configured according to the key parameters of the actual visual inspection camera (including focal length, field of view, resolution, etc.). This step ensures that the camera in the twin system is consistent with the actual camera in performance, making the output images and data closer to the real world. In addition, the number and position of cameras in the twin system can be flexibly adjusted according to the actual application scenario, fully covering the detailed attention to different distances and angles to achieve complete correspondence with the real world.

[0016] Second, industrial scene modeling and lighting simulation. In the digital twin system, accurate modeling and scene construction are carried out according to the actual industrial scene, providing a variety of different industrial scenes to meet training needs. The twin system can also freely set lighting conditions to truly simulate the color and intensity of the light on site and enhance the realism of the simulated environment.

[0017] Third, customize digital characters and simulate their movements. In the setting of digital characters, make detailed adjustments to the height, body shape, gender, age, and clothing of digital characters to ensure that the character image is closer to reality and increase the diversity and richness of the characters. Obtain the data in the annotation file output by the data set or visual detection, import it into the digital twin system, and write the corresponding code so that the characters in the twin system can move according to the settings, and adjust the data input interval to ensure the continuity and smoothness of the character's movement. In view of the occlusion phenomenon between people and people, and people and objects in the real world, write special code in the twin system and make corresponding settings to ensure the correct judgment of the occlusion relationship.

[0018] Fourth, automatic output and identification of training data. The twin system outputs the annotated files and pictures required for machine vision training, where the annotated files include the character's identity feature description (such as height, body shape, gender, age, occupation, etc.), joint data, joint occlusion relationship, character range box, scene features, screen range, and camera's internal and external parameter matrix and other required information. By writing specific code, the twin system can automatically generate this data and allow users to adjust the output time interval to optimize output efficiency. During the data output process, parameters such as scene settings, camera viewing angles, character models and their wearing characteristics can be adjusted autonomously or automatically changed to ensure that the output data set is richer and more diverse in content. In addition, the system can automatically identify and filter out actions with high similarity by calculating the similarity of the previous and subsequent actions, while ensuring that the differences between the actions are not too large, which helps to improve the generalization ability of the data set.

[0019] The beneficial effects of the present invention are:

[0020] This invention solves the problem of lack of human posture simulation data set training in industrial scenes under digital twin environment, and proposes a method for automatically generating and identifying human posture multi-view visual recognition AI training data set based on simulation environment. This method uses digital twin technology, synchronizes real camera settings, simulates real industrial scenes and lighting conditions, simulates human movements, and automatically generates information-rich annotation files and pictures, providing a large number of high-quality and efficient data sets for human posture machine vision AI training in industrial scenes. This method has the following significant advantages:

[0021] First of all, this method can construct a variety of different industrial scenes, create diverse human models, simulate rich human movements, set different lighting conditions, change the number, distance and angle of cameras, etc. These factors can be freely combined, making the data set content richer and the amount of data increasing, ensuring the robustness, scalability and generalization ability of the data set, and effectively solving the problem of difficulty in obtaining human posture simulation data sets in industrial scenes.

[0022] Second, the data set generated by digital twin technology is no less authentic than the existing data set. In the twin environment, the industrial environment and its lighting conditions are restored according to the real scene, a rich and diverse human body model is customized, and human body movements are simulated smoothly and coherently. These steps make the data set closer to the real world and enhance its applicability in machine vision recognition AI training.

[0023] Third, the digital twin system can automatically output the labeled data of human joints, occlusion relationships, human range frames, screen ranges and other information, and automatically identify and filter out actions with high similarity by calculating similarity when outputting data, while ensuring that the differences between actions are not too large. This optimized data set not only improves the accuracy and processing efficiency of the data, but also significantly improves the performance of the machine learning model compared to traditional methods, ensuring the adaptability and reliability of the model in practical applications.

[0024] Fourth, in the twin system, the digital human moves according to the data output by visual detection, and outputs more accurate key point motion data. This data can be input into the twin system again to make the digital human move and iterate the data. In the twin system, the output data accuracy can be selected. Through this iterative process, the accuracy of the digital human motion data is further improved.

[0025] Fifth, digital twin technology reduces dependence on physical experiments and on-site data collection through virtual simulation, thereby saving a lot of time and money, obtaining high-quality, high-quantity and high-efficiency data sets at a lower cost, and accelerating the development of industrial visual recognition training. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Illustration of some industrial scenes in the twin system.

[0027] Figure 2 Schematic diagram of the digital human skeleton in the twin software.

[0028] Figure 3 Schematic diagram of digital human motion data iteration in the digital twin environment.

[0029] Figure 4 Schematic diagram of judging the occlusion relationship of a digital human in a digital twin environment.

[0030] Figure 5 Skeleton joint point representation diagram during similarity calculation process.

[0031] Figure 6 Schematic diagram of angle calculation during similarity calculation.

[0032] Figure 7 Similarity algorithm flow chart.

[0033] Figure 8 Flowchart of the method for automatic generation and identification of AI training data in a simulation environment for human-machine posture machine vision recognition. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0035] like Figure 1-8 shown.

[0036] A method for automatically generating and identifying simulation environment AI training data for human-machine posture machine vision recognition, the implementation of which includes the following steps:

[0037] Step 1: For the camera settings in the digital twin system, first configure the internal parameters and external parameters such as position and angle based on the relevant information of the actual visual inspection camera to ensure that the camera in the digital twin environment can accurately simulate and reflect the characteristics of the actual camera. The twin camera turns on the physical camera settings. The physical camera parameters include focal length, field of view, sensor type and size, lens offset, etc. These internal parameter settings can better simulate the camera effect of the real camera.

[0038] In addition, the number and position of cameras in the digital twin system can also be flexibly adjusted according to the needs of the actual application scenario. This adjustment helps to ensure the authenticity and accuracy of the output image, so that the digital twin system can provide more reliable data support when simulating industrial scenarios. For easy management, all cameras in the twin system are placed under a common parent object. This layout not only allows a single camera to move and rotate independently, but also greatly simplifies the overall layout and adjustment of the camera group. In the process of outputting the data set, the position and angle of the camera can be adjusted to make its shooting range and viewing angle more diverse, more realistically simulate the camera state in the real scene, and enrich the image content in the data set.

[0039] When conducting machine vision deep learning, especially human posture recognition, it is crucial to understand the camera's internal and external parameter matrices and other related parameters. These parameters play a decisive role in the geometric distortion correction and depth information recovery of the image, and directly affect the accuracy of posture recognition and the effect of model training. In order to better support human posture machine vision recognition, the annotation file content output by the twin dataset must include the camera's internal and external parameter matrices.

[0040] The intrinsic parameter matrix of a camera is used to describe the internal parameters of the camera, including the focal length of the camera, the coordinates of the principal point, and the distortion of the image. The intrinsic parameter matrix is ​​usually in the form of a 3x3 matrix, commonly represented as K. The intrinsic parameter matrix can map the three-dimensional points in the camera coordinate system to the two-dimensional pixel coordinates on the image plane. Through the intrinsic parameter matrix, operations such as camera calibration, image correction, and projection of three-dimensional point clouds to images can be performed. The expression of the intrinsic parameter matrix is:

[0041]

[0042] Among them, f x 、f y Use pixels to describe the length of the focal length in the x and y axes, (C x , C y ) represents the coordinates of the center point of the image, which specifies the origin of the image plane and is used to remove the translation distortion of the image. In the twin system, given the focal length f of the camera, the sensor size, and the field of view (FOV) of the camera, f can be calculated. x 、f y First, calculate the horizontal and vertical field of view FOV of the camera x and FOV y ; Then, f can be calculated by the following formula x 、f y .

[0043]

[0044] Where imageWidth and imageHeight are the resolution of the image in pixels, S x , S y is the physical size of the sensor in millimeters. C x , C y The center point is calculated by obtaining the image size.

[0045] The camera's extrinsic matrix is ​​a parameter used to describe the position and posture of the camera in the world coordinate system. It is usually in the form of a 4x4 matrix, including a 3x3 rotation matrix and a 3x1 translation vector. The rotation matrix represents the rotation of the camera coordinate system relative to the world coordinate system, and each column represents the representation of an axis (X, Y, Z axis) in the camera coordinate system in the world coordinate system. The translation vector represents the position of the origin of the camera coordinate system in the world coordinate system. These two elements together define the position and posture of the camera coordinate system relative to the world coordinate system. The expression of the extrinsic matrix is:

[0046]

[0047] Simplified expression:

[0048]

[0049] Where R is the rotation matrix and t is the translation vector. In a twin system, a 4x4 matrix can be constructed to obtain the rotation matrix and translation vector of the camera for combination.

[0050] Step 2: Accurately model and build scenarios in the twin system based on actual industrial scenarios. For scenarios involving confidentiality, the models are desensitized before use to ensure data security. The models in the twin system are guaranteed to be consistent with the actual scenarios in terms of color, size, position, and lighting changes.

[0051] In the twin system, materials and shaders are used to achieve the desired object appearance display effect. Materials are used to describe the surface details of 3D objects, including color, smoothness, transparency, metal material settings, etc.; shaders define the various codes (such as vertex shaders and fragment shaders), attributes (such as textures, etc.) and instructions (rendering and label settings, etc.) required for rendering. The process includes creating materials, creating shaders, assigning shaders to the materials created in the previous step, then assigning materials to objects to be rendered, and adjusting the properties of shaders in the material panel to obtain satisfactory results. Through the combined use of materials and shaders, the digital twin system can provide a more realistic, flexible and efficient rendering solution, which helps to improve the accuracy and authenticity of the simulation. Figure 1 This is a demonstration of some industrial scenarios in the twin system.

[0052] Setting up lighting in the twin system is a key step to achieve realistic rendering effects. The twin system provides a variety of light source types and lighting settings. The following are the implementation steps: First, add a light source. The twin system supports three types of light sources: directional light that simulates sunlight, parallel light, point light sources that simulate emitting light from a point in all directions, and spotlights that simulate emitting conical light from a point; secondly, adjust the light source properties, set the color, brightness, range of influence, light attenuation, shadows, etc. of the light source; then, configure the lighting mode. The twin system provides two lighting modes: real-time global illumination and baked global illumination. Select and set them according to the actual situation; finally, adjust the material, use lighting probes, set reflections, optimize lighting performance, etc. according to the situation. Through these steps, you can create a variety of lighting effects in the twin system to meet different needs and be closer to the real world.

[0053] In the process of outputting the data set, write the corresponding code (which can be automatically generated by the system or written manually, and can be implemented using existing technologies, the same below) and perform related operations to automatically or manually set the lighting effects in the scene to comply with the natural laws of lighting transformation. At the same time, the appearance of the model can be changed by changing the material and shader, or switching different industrial scenes to enrich the scene settings in the data set and enhance the diversity and practicality of the data set.

[0054] Step 3: Customize the digital human model and simulate motion in the digital twin system. In the twin system, the digital human consists of skin and skeleton. The skin covers the skeleton and is responsible for shaping the shape and appearance of the digital human, while the skeleton is responsible for driving the movement of the digital human. Through the precise movement of the skeleton, the skin will deform accordingly, giving the digital human a lifelike image and smooth dynamic performance.

[0055] The digital human appearance settings include height, body type, gender, age, clothing, etc., and the material and shader settings are used to ensure that the character image is more realistic. In the process of outputting the data set, the digital human appearance can be transformed according to the written code and the changes in the material and shader, enriching the character image in the data set.

[0056] A digital human model with bones can be directly imported into the twin system, but it is necessary to ensure that the bone points of the model match the bone points predefined in the twin system. The bones are organized in a tree-like hierarchy, such as Figure 2 As shown in the figure, there is a root bone in the entire skeleton structure, and other bones are directly or indirectly connected to the root bone to form the entire skeleton framework of the character model. Generally, each bone has two matrices, one is the initial transformation matrix to represent the initial position of the bone; the other is the combined transformation matrix used to perform various transformations on the bone to achieve character movement.

[0057] In the twin system, the coordinate value changes of each joint of the digital human are used to represent the translation transformation of the digital human, and the rotation transformation of the digital human is controlled and represented by quaternions. Quaternions are a mathematical tool used to represent rotations in three-dimensional space. They consist of a real part and three imaginary parts in the form of w+xi+yj+zk, where w is the real part and x, y, and z are the imaginary parts. The main advantage of quaternions is that they can represent three-dimensional rotations concisely and efficiently, and avoid the problem of universal joint lock, ensuring the accuracy and stability of physical simulation.

[0058] Digital human motion simulation is a key step in this method. By reproducing the actions in the input file, the digital human can move smoothly in the twin environment, and finally output accurate joint motion data, converting the low-accuracy data in the input file into high-accuracy data to meet the accuracy requirements of the industrial data set. At the same time, by continuously improving the accuracy requirements, the data set iteration is completed. The flow chart is shown in the figure. Figure 3 shown.

[0059] Taking the twin software Unity as an example, the specific steps are as follows:

[0060] (3.1) Use the SQLite4Unity3d plug-in to obtain the data in the annotation file output by the dataset or visual detection, and import it into the twin system. The annotation file includes the position data of each key point of the digital human and the quaternion data between the key points;

[0061] (3.2) Match the key point data in the file with the joints of the digital human, and drive the movement of the digital human through the quaternion rotation Quaternion.Slerp (QuaternionfromRotation, QuaterniontoRotation, float) to ensure the continuity and smoothness of the rotation, where fromRotation is the starting quaternion, toRotation is the ending quaternion, and t is the interpolation ratio, and the value range is [0,1]. In order to ensure that the movements of the digital human are smooth and natural, the movement effect can be optimized by adjusting the time interval of data input. Note that not all nodes are equipped with quaternions; quaternions are only applied when there are child objects at the joint points, because in the digital twin system, the movement of the parent object will automatically drive its child objects, and the movement of the child object will not affect the parent object;

[0062] (3.3) During the movement of digital humans, there will be mutual occlusion and occlusion by objects. It is necessary to judge the occlusion situation in order to output it to the annotation file. The flowchart for judging the occlusion relationship is as follows Figure 4 shown.

[0063] In Unity, a ray is emitted to detect whether an object has a collider or trigger. Objects without collision components cannot be detected. The detection function is

[0064] Physics.Raycast(Rayray,outhit,floatmaxDistance,layerMask), where ray is a ray whose launch position and launch direction can be set, hit is the information of the object hit by the ray, maxDistance is the ray distance, and the default indicates infinite distance, layerMask is the ray mask, which indicates which layer is detected, and the default indicates detection of all layers. In view of the actual situation, the work in factory workshops is mostly completed by multiple people, so different numbers of multi-person movements often appear in twin scenes. Multi-person collision detection includes people-to-people and people-to-objects. It is necessary to detect each joint of each person, so each person is on a different layer and the object is on the same layer, and colliders need to be set for the digital human skin and objects to ensure that the ray can be detected normally. The judgment of the occlusion relationship can be simplified to the intersection test between the ray and the object, and the detection judgment formula is as follows:

[0065] Ray=StartPoint+t·Direction

[0066] Among them, StartPoint is the starting point of the ray, Direction is the direction of the ray (needs to be standardized), and t is the distance from the starting point to the intersection point. If t is between 0 and 1, it means that the ray intersects with the object between the starting point and the end point, that is, there is occlusion. If t is equal to 1, it means that the ray successfully detects the end point, that is, there is no occlusion. The occlusion relationship of the dataset is expressed as follows: 1 for occluded joints and 2 for unoccluded joints, which is expressed as

[0067]

[0068] (3.4) After the digital human moves smoothly, the digital human's clothing is changed manually or automatically through code writing and changes in materials and shaders, and the appropriate time interval and accuracy are set to output the coordinates and occlusion relationship of each joint point of the digital human. This process not only provides important data support for the multi-person key point detection algorithm, but also realizes the data set iteration process through the continuous improvement of output accuracy. By analyzing the digital human joint data, we can determine the maximum and minimum values ​​of the data on the X, Y, and Z axes of the character, and then obtain the character's range box, and output it to the annotation file, which provides great convenience for the implementation of the multi-person detection algorithm.

[0069] The multi-person key point detection algorithm used in the current data set is mostly a top-down key point detection method. This method first determines the human body range of a single person, and then performs key point detection on the single person within the image detection frame, converting the multi-person key point detection problem into a single-person key point detection problem. Although this method performs well in terms of key point accuracy, its processing speed is relatively slow and it is easy to miss or re-check. Therefore, in the twin software, the human body range frame is quickly calculated through more accurate data, which not only solves the problem of missed detection or false detection that is very easy to occur when the human bodies are mutually occluded or the postures are complex, but also greatly improves the processing efficiency.

[0070] In the fourth step, the twin system automatically outputs the annotated files and pictures required for machine vision training. The annotated files include detailed information such as the character's identity feature description (such as height, body shape, gender, age, occupation, etc.), joint data, joint occlusion relationship, character range box, scene features, screen range, and camera internal and external parameter matrices. Annotated files and pictures can be automatically generated by writing corresponding codes through the twin system, and the output time interval can be changed to improve output efficiency. During the output process, the scene, camera perspective, character model and its clothing characteristics can be modified autonomously or automatically, making the output data set content richer and more diverse.

[0071] In the field of machine vision recognition, similarity detection of digital human actions is a key task. If there are a large number of actions with extremely high similarity in the dataset, it may cause the model to overfit during the machine vision recognition process. In this case, the model may focus too much on the detailed features of these specific actions and ignore the broader, generalized action features, which will affect the performance and generalization ability of the model on new data. On the contrary, if the actions in the dataset are too different, the model may not be able to learn effectively due to the lack of sufficient feature information, resulting in underfitting. In this case, the performance of the model on both the training set and the test set may be unsatisfactory because it cannot capture enough features that can represent all actions. Therefore, in order to improve the generalization ability and performance of the model, it is necessary to balance the similarity and difference of actions in the dataset output and model training, and perform similarity detection during the dataset output process to ensure that the model can learn both specific and generalized features.

[0072] In the output process, the similarity detection uses the cumulative sum of the angles of the previous and next actions to make a judgment. The smaller the weighted cumulative sum of the angles of the previous and next actions, the higher the similarity of the two actions. There are 19 skeletal joints in total, such as Figure 5 As shown, the information is recorded as J, which is expressed as J = (j0, j1, ..., j 18 ), the position information of each joint point includes three dimensions, denoted as j i , expressed as:

[0073] j i =(x i ,y i , z i ),i∈[0,18].

[0074] Since the lengths of the digital human skeleton segments are inconsistent, the weight of each joint point should be different when calculating the similarity, so a weight-adaptive skeleton similarity algorithm is proposed. The specific process is:

[0075] (4.1) Planning skeleton segments, there are 18 skeleton segments with 19 joints. Since the order of the starting and ending points of the skeleton segments affects the algorithm results, the starting and ending points of each segment are fixed. The starting joint point of each segment is numbered as s, the ending joint point is numbered as e, k represents the kth segment, and the length is l k (p s ,p e ),k∈

[0076] [0,17],s∈[0,18],e∈[0,18], the detailed information of the start and end points of the skeleton segment is shown in Table 1.

[0077] l k (p s ,p e ) is calculated as:

[0078]

[0079] Table 1 Numbering of the starting and ending points of each line segment of the skeleton

[0080]

[0081] (4.1) Define the weight W of the joint points. There are 19 weights corresponding to 19 joints: W = (w0, w1, w2, …, w 17 ,w 18 ).

[0082] (4.2) Weight adaptive value selection. The value selection principle is that the tail joint point of the longer bone edge in the template information has a larger corresponding weight, and the tail joint point of the corresponding shorter bone edge has a smaller weight. If the hip joint is selected as the reference point, its corresponding weight is 0, and the joint coordinates of the previous action are For example, specifically:

[0083]

[0084] (4.3) Calculate the action similarity. First, use the vector coordinates to calculate the previous action and the next action The angle of the corresponding bone Figure 6 This is a schematic diagram of angle calculation.

[0085]

[0086] Then calculate the total angle θ(P,Q) according to the above weights for each angle value:

[0087]

[0088] The weighted total angle θ(P,Q) is a number greater than or equal to 0, and the larger the value, the smaller the corresponding action similarity. When the weighted total angle is large, it means that the similarity between the previous and next actions is low. This may be because the interval between action outputs is too large, resulting in significant differences between actions. In this case, you can consider shortening the interval between action outputs to reduce the differences between actions. On the contrary, when the weighted total angle is small, it indicates that the similarity between the previous and next actions is high. This may be because the interval between action outputs is too small or the amplitude of the character's action is too small. At this time, you can appropriately increase the interval between action outputs, or directly skip the current action and select the next action for output. The similarity algorithm flow chart is as follows: Figure 7 shown.

[0089] The present invention does not involve parts that are the same as the prior art or can be implemented by using the prior art.

Claims

1. A method for automatically generating and identifying AI training data sets for multi-view human posture recognition based on a simulation environment, characterized by: First, camera parameters are matched; the camera in the digital twin system is configured according to the key parameters of the actual visual inspection camera; this ensures that the camera in the twin system is consistent with the actual camera in performance, making the output images and data closer to the real world; In addition, the number and position of cameras in the twin system should be flexibly adjusted according to the actual application scenarios, fully covering the detailed attention to different distances and angles to achieve full correspondence with the real world; Second, industrial scene modeling and lighting simulation: In the digital twin system, accurate modeling and scene construction are carried out according to the actual industrial scene, providing a variety of different industrial scenes to meet training needs; the twin system can also freely set lighting conditions to truly simulate the color and intensity of the light on site, enhancing the realism of the simulated environment; Third, customize digital characters and simulate their movements. In terms of setting up digital characters, make detailed adjustments to the height, body shape, gender, age, and clothing of digital characters to ensure that the characters are closer to reality and increase the diversity of characters. Obtain data from the annotation files of data sets or visual detection outputs, import them into the digital twin system, and write corresponding codes so that the characters in the twin system can move according to the settings. At the same time, adjust the data input interval to ensure the continuity and smoothness of the characters' movements. In view of the occlusion phenomenon between characters and characters, and between characters and objects in the real world, write special codes in the twin system and make corresponding settings to ensure the correct judgment of the occlusion relationship. Fourth, the training data is automatically output and labeled; the annotated files and pictures required for machine vision training are output through the twin system, where the annotated files include the character's identity feature description, joint data, joint occlusion relationship, character range box, scene features, screen range, and the information required for the camera's internal and external parameter matrix; by writing specific code, the twin system can automatically generate this data and allow users to adjust the output time interval to optimize output efficiency; during the data output process, scene settings, camera perspectives, character models and their clothing features can be adjusted autonomously or automatically changed to ensure that the output data set is richer and more diverse in content; in addition, the system can automatically identify and filter out actions with higher similarity by calculating the similarity between previous and subsequent actions, while ensuring that the differences between actions are not too large, which helps to improve the generalization ability of the data set.

2. The method according to claim 1, characterized in that: The key parameters of the camera include focal length, field of view and resolution.

3. The method according to claim 1, characterized in that: The character's identity characteristics description includes height, body shape, gender, age and occupation.

4. The method according to claim 1, characterized in that: The number and position of cameras in the digital twin system should be flexibly adjusted according to the needs of the actual application scenario. This adjustment helps to ensure the authenticity and accuracy of the output image, so that the digital twin system can provide more reliable data support when simulating industrial scenarios. For the convenience of management, all cameras in the twin system are placed under a common parent object. This layout not only allows a single camera to move and rotate independently, but also greatly simplifies the overall layout and adjustment of the camera group. In the process of outputting the data set, the position and angle of the camera should be adjusted to make its shooting range and viewing angle more diverse, more realistically simulate the camera state in the real scene, and enrich the picture content in the data set. In deep learning of machine vision, especially Especially when recognizing human posture, understanding the camera's internal and external parameter matrices plays a decisive role in the geometric distortion correction and depth information recovery of the image, which directly affects the accuracy of posture recognition and the effect of model training. To better support human posture machine vision recognition, the annotation file content output by the twin data set must contain the camera's internal parameter matrix and external parameter matrix. The camera's internal parameter matrix is ​​used to describe the camera's internal parameters, including the camera's focal length, principal point coordinates, and image distortion information. The internal parameter matrix is ​​in the form of a 3x3 matrix, denoted as K. The internal parameter matrix maps the three-dimensional points in the camera coordinate system to the two-dimensional pixel coordinates on the image plane. The internal parameter matrix can be used for camera calibration, image correction, and projection of three-dimensional point clouds to images. The expression of the internal parameter matrix is: Among them, f x 、f y Use pixels to describe the length of the focal length in the x and y axes, (C x , C y ) represents the coordinates of the center point of the image, which specifies the origin of the image plane and is used to remove the translation distortion of the image. In the twin system, the focal length f of the camera, the sensor size, and the field of view (FOV) of the camera are known to calculate f x 、f y : First, calculate the horizontal and vertical field of view FOV of the camera x and FOV y ; Then, calculate f by the following formula x 、f y ; Where imageWidth and imageHeight are the resolution of the image in pixels, S x , S y is the physical size of the sensor in millimeters, C x , C y It is obtained by calculating the center point by obtaining the image size; The camera's extrinsic matrix is ​​a parameter used to describe the position and posture of the camera in the world coordinate system. It is usually in the form of a 4x4 matrix, including a 3x3 rotation matrix and a 3x1 translation vector. The rotation matrix represents the rotation of the camera coordinate system relative to the world coordinate system, and each column represents the representation of one of the X, Y, and Z axes in the camera coordinate system in the world coordinate system. The translation vector represents the position of the origin of the camera coordinate system in the world coordinate system. These two elements together define the position and posture of the camera coordinate system relative to the world coordinate system. The expression of the extrinsic matrix is: Simplified expression: Among them, R is the rotation matrix and t is the translation vector. In the twin system, a 4x4 matrix is ​​constructed to obtain the rotation matrix and translation vector of the camera for combination.

5. The method according to claim 1, characterized in that: When modeling the industrial scenarios described above, for scenes involving confidentiality, the models should be desensitized before use to ensure data security; the models in the twin system should ensure that their color, size, position, and lighting changes are consistent with the actual scene; in the twin system, the required object appearance display effect is achieved through materials and shaders; materials are used to describe the surface details of 3D objects, including color, smoothness, transparency, and metal material settings; shaders define the various codes, properties, and instructions required for rendering; the process includes creating materials, creating shaders, and assigning shaders to the materials created in the previous step; then assigning materials to the objects that need to be rendered For objects to be rendered, adjust the properties of the Shader in the material panel to obtain satisfactory results; through the combined use of materials and Shaders, the digital twin system can provide a more realistic, flexible and efficient rendering solution, which helps to improve the accuracy and authenticity of the simulation; setting up lighting simulation in the twin system is a key step in achieving realistic rendering effects; the twin system provides a variety of light source types and lighting settings. The following are the implementation steps: First, add a light source. The twin system supports three types of light sources: directional light that simulates sunlight, parallel light, point light sources that simulate emitting light from a point in all directions, and spotlights that simulate emitting conical light from a point; Secondly, adjust the light source properties and set the color, brightness, influence range, light attenuation, and shadow of the light source; then, configure the lighting mode. The twin system provides two lighting modes: real-time global illumination and baked global illumination. Select and set them according to the actual situation; finally, adjust the material, use light probes, set reflections, and optimize lighting performance according to the situation; in the process of outputting the data set, write the corresponding code and perform related operations to automatically or manually set the lighting effects in the scene to comply with the natural laws of lighting transformation; at the same time, the appearance of the model can be changed by changing the material and shader, or switching between different industrial scenes to enrich the scene settings in the data set and enhance the diversity and practicality of the data set.

6. The method according to claim 1, characterized in that: In the twin system, the digital human is composed of skin and skeleton. The skin covers the skeleton and is responsible for shaping the shape and appearance of the digital human, while the skeleton is responsible for driving the movement of the digital human. Through the precise movement of the skeleton, the skin will produce corresponding deformation, thus giving the digital human a lifelike image and smooth dynamic performance. The digital human appearance settings include height, body shape, gender, age, and clothing. The material and shader settings are used to ensure that the character image is more realistic. In the process of outputting the data set, the digital human appearance should be transformed according to the written code and the changes in the material and shader to enrich the character image in the data set. The digital human model with bones can be directly imported into the twin system, but it is necessary to ensure that the bone points of the model match the bone points predefined by the twin system. The bones are organized in a tree-like hierarchical structure. There is a root bone in the entire bone structure, and other bones are directly or indirectly connected to the root bone to form the entire bone framework of the character model. Each bone has two There are two matrices, one is the initial transformation matrix that represents the initial position of the bones; the other is the transformation matrix used to transform and combine the bones to achieve character movement; in the twin system, the coordinate value changes of each joint of the digital human are used to represent the translation transformation of the digital human, and quaternions are used to control and represent the rotation transformation of the digital human; quaternion is a mathematical tool used to represent rotation in three-dimensional space, which consists of a real part and three imaginary parts, in the form of w+xi+yj+zk, where w is the real part, x, y, z are imaginary parts; the main advantage of quaternion is that it can represent three-dimensional rotation concisely and efficiently, and avoid the problem of universal joint lock, ensuring the accuracy and stability of physical simulation.

7. The method according to claim 1, characterized in that: The digital human motion simulation reproduces the actions in the input file, so that the digital human can move smoothly in the twin environment, and finally outputs accurate joint motion data, converting the low-accuracy data in the input file into high-accuracy data to meet the accuracy requirements of the industrial data set. At the same time, the data set iteration is completed by continuously improving the accuracy requirements. The specific steps are as follows: 7.1 Use the SQLite4Unity3d plug-in to obtain the data in the annotation file output by the dataset or visual detection, and import it into the twin system; the annotation file includes the position data of each key point of the digital human and the quaternion data between the key points; 7.2 Match the key point data in the file with the joints of the digital human, and drive the digital human to move through the quaternion rotation Quaternion.Slerp (Quaternion fromRotation, Quaternion toRotation, float t) to ensure the continuity and smoothness of the rotation, where fromRotation is the starting quaternion, toRotation is the ending quaternion, and t is the interpolation ratio, with a value range of [0,1]. In order to ensure the smooth and natural movements of the digital human, the movement effect is optimized by adjusting the time interval of data input; quaternions are only applied when there are child objects at the joint points, because in the digital twin system, the movement of the parent object will automatically drive its child objects, while the movement of the child object will not affect the parent object; 7.3 During the movement of digital humans, there may be mutual occlusion or occlusion by objects. It is necessary to judge the occlusion situation in order to output it into the annotation file; In Unity, a ray is emitted to detect whether the object has a collider or trigger. Objects without collision components cannot be detected. The detection function is Physics.Raycast(Ray ray,out hit,floatmaxDistance,layerMask), where ray is a ray, and its emission position and emission direction can be set. hit is the information of the object hit by the ray. maxDistance is the ray distance, which means infinite distance by default. layerMask is the ray mask, which indicates which layer is detected. By default, it means detecting all layers. In view of the actual situation, the work in factory workshops is mostly completed by multiple people, so different numbers of multi-person movements often appear in twin scenes. Multi-person collision detection includes people and people, people and objects, and each joint of each person needs to be detected. Therefore, each person is on a different layer, and the object is on the same layer. In addition, colliders need to be set for the digital human skin and objects to ensure that the ray can be detected normally. The judgment of the occlusion relationship can be simplified to the intersection test between the ray and the object. The detection judgment formula is as follows: Ray=StartPoint+t·Direction Among them, StartPoint is the starting point of the ray, Direction is the direction of the ray that needs to be standardized, and t is the distance from the starting point to the intersection point; if t is between 0 and 1, it means that the ray intersects with the object between the starting point and the end point, that is, there is occlusion; if t is equal to 1, it means that the ray successfully detects the end point, that is, there is no occlusion; the occlusion relationship of the data set is expressed as, the occluded joint is represented as 1, and the unoccluded joint is represented as 2, which is expressed as 7.4 After the digital human moves smoothly, the digital human's clothing is changed manually or automatically through code writing and changes in materials and shaders, and the appropriate time interval and accuracy are set to output the coordinates and occlusion relationships of each joint point of the digital human. This process not only provides important data support for the multi-person key point detection algorithm, but also realizes the data set iteration process by continuously improving the output accuracy; by analyzing the digital human joint data, the maximum and minimum values ​​of the person's data on the X, Y, and Z axes are determined, and then the person's range frame is obtained, and it is output to the annotation file, which provides great convenience for the implementation of the multi-person detection algorithm; the multi-person key point detection algorithm used in the current data set is mostly a top-down key point detection method. This method first determines the human body range of a single person, and then performs key point detection on the single person in the image detection frame, converting the multi-person key point detection problem into a single person key point detection problem.

8. The method according to claim 1, characterized in that: When performing similarity detection on the operation of digital humans, if there are a large number of highly similar actions in the data set, it will cause the model to overfit during the machine vision recognition process; in this case, the model will focus too much on the detailed features of these specific actions and ignore the broader, generalized action features, thus affecting the model's performance and generalization ability on new data; on the contrary, if the differences in the actions in the data set are too large, the model will not be able to learn effectively due to the lack of sufficient feature information, resulting in underfitting; in this case, the model's performance on both the training set and the test set may be unsatisfactory because it cannot capture enough features that can represent all actions. Therefore, in order to improve the generalization ability and performance of the model, it is necessary to balance the similarity and difference of actions in the data set output and model training, and perform similarity detection during the data set output process to ensure that the model can learn both specific and generalized features; In the output process, similarity detection uses the cumulative sum of the angles of the previous and next actions for judgment; the smaller the weighted cumulative sum of the angles of the previous and next actions, the higher the similarity of the two actions; there are 19 skeletal joints in total, and their information is recorded as J, expressed as J = (j0, j1, ..., j 18 ), the position information of each joint point includes three dimensions, denoted as j i , expressed as: j i (x i y i ,z i ), i∈[0.18]: Since the lengths of the digital human skeleton segments are inconsistent, the weight of each joint point should be different when calculating the similarity. Therefore, a weight-adaptive skeleton similarity algorithm is proposed. The specific process is as follows: 8.1 Planning skeleton segments, there are 18 skeleton segments with 19 joints. Since the order of the starting and ending points of the skeleton segments affects the algorithm results, the starting and ending points of each segment are fixed. The starting joint point of each segment is numbered as s, the ending joint point is numbered as e, k represents the kth segment, and the length is l k (p s ,p e ),k∈ [0,17],s∈[0,18],e∈[0,18];l k (p s ,p e ) is calculated as: 8.2 Define the weight W of the joint points. There are 19 weights corresponding to 19 joints: In=(in0,in1,in2,…,in 17 ,In 18 ); 8.3 Weight adaptive value selection; the value selection principle is that the tail joint point of the longer bone edge in the template information has a larger corresponding weight, and the tail joint point of the corresponding shorter bone edge has a smaller weight. If the hip joint is selected as the reference point, its corresponding weight is 0, and the joint coordinates of the previous action are For example, specifically: 8.4 Calculate action similarity; first use vector coordinates to calculate the previous action and the next action The angle of the corresponding bone Then calculate the total angle θ(P,Q) according to the above weights for each angle value: The weighted total angle θ(P,Q) is a number greater than or equal to 0, and the larger the value, the smaller the corresponding action similarity; when the weighted total angle is large, it means that the similarity between the previous and next actions is low, because the interval of action output is too large, resulting in significant differences between actions; in this case, shortening the interval of action output should be considered to reduce the difference between actions; on the contrary, when the weighted total angle is small, it indicates that the similarity between the previous and next actions is high, because the interval of action output is too small or the amplitude of the character's action is too small. At this time, the interval of action output should be appropriately increased, or the current action should be skipped directly and the next action should be selected for output.

Images