Method for constructing a training dataset for training an action recognition model

Abstract

The present application relates to the technical field of data set construction, and specifically discloses a method for constructing a training data set for training an action recognition model. The present application can use a large action generation model to generate a target action instance according to actual needs, obtain an initial data set according to the target action instance, and use a large visual model to generalize the initial data set to generate an augmented data set as target training data, thereby improving the use value of data in the field of AI and solving the defects of poor generalization, small scale and high generation cost of traditional simulation data; and the generation method does not rely on manually labeled data, avoiding problems such as incorrect labeling and inconsistent labeling caused by manual experience errors, solving the defects of difficult scale, poor accuracy, high cost and low efficiency of manually labeled data, and improving the data scale, accuracy, data generalization and construction efficiency of the data set.

Description

Technical Field

[0001] The present application relates to the technical field of data set construction, and in particular to a method for constructing a training data set for training an action recognition model. Background Art

[0002] With the rapid development of science and technology, AI (Artificial Intelligence) has become the core of innovation. The development of machine learning and deep learning methods is the key link for AI technology to enter the industry and promote technological change. Among them, supervised machine learning methods represented by deep learning rely on a large number of annotated high-quality data sets, which are the cornerstone of training and evaluating AI models.

[0003] At present, various professional sub-fields are building their own large-scale data sets for deep learning, including large-scale mechanized data collection, large-scale data factories for manual labeling, and large-scale simulation data set generation, such as the autonomous driving industry. Among them, the visual problem of behavior recognition, as a subdivided basic technology field for current human-computer interaction, digital twins, and metaverse applications, is far more difficult to build a data set than most other AI fields. Because behavioral actions are difficult to collect in the real world, large-scale manual labeling has poor accuracy, and the high data dimension leads to a huge workload. Humans cannot even accurately and efficiently label spatial data; on the other hand, although the simulation field can generate accurately labeled behavioral action data, due to the high data dimension, it is currently difficult to generate reasonable data with good generalization performance in the simulation field. It is often limited to a small number of simulated rendering outputs of fixed actions, and the data has a high degree of duplication, making it difficult to promote on a large scale in AI training.

[0004] The above problems have not only limited the development of related machine learning methods, but also limited the speed and cost of industrial application development in the field of behavior recognition. Therefore, how to improve the data size, annotation accuracy, data generalization and construction efficiency of behavior recognition data sets has become an urgent problem that has extremely high practical value. Summary of the invention

[0005] The present application provides a method for constructing a training data set for training an action recognition model, aiming to efficiently generate high-quality, large-scale behavior recognition data with the help of visual big model technology, thereby improving the data scale, annotation accuracy, data generalization and construction efficiency of data sets generated by traditional manual or simulation methods, so that the constructed data set has extremely high use value in machine learning training corresponding to behavior recognition AI applications.

[0006] In a first aspect, the present application provides a method for constructing a training data set for training an action recognition model, the method comprising:

[0007] Generate a large model based on the preset action, process the preset action description information, and generate a target action instance;

[0008] Based on the target action instance, a scene animation is constructed, and the scene animation is processed to obtain target parameter data, and based on the target parameter data, an initial data set is generated;

[0009] The initial data set is processed to obtain model parameters, and a large model and the model parameters are generated based on a preset image to generate an augmented data set as a target training data set.

[0010] In a second aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the method of constructing a training data set for training an action recognition model as described above when executing the computer program.

[0011] In a third aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the method of constructing a training data set for training an action recognition model as described above.

[0012] The present application discloses a method for constructing a training data set for training an action recognition model, generating a large model based on a preset action, processing the preset action description information, and generating a target action instance; based on the target action instance, constructing a scene animation, and processing the scene animation to obtain target parameter data, and generating an initial data set based on the target parameter data; processing the initial data set to obtain model parameters, and generating an augmented data set based on a preset image generation large model and the model parameters, as a target training data set. The present application can generate a target action instance according to actual needs, obtain an initial data set according to the target action instance, and generalize the initial data set with the help of a visual large model to generate an augmented data set as target training data, thereby improving the use value of data in the field of AI, and solving the defects of poor generalization, small scale and high generation cost of traditional simulation data; and the generation method does not rely on manually labeled data to avoid problems such as incorrect labeling and inconsistent labeling caused by human experience errors, and solves the defects of difficulty in scaling up, poor accuracy, high cost and low efficiency of manually labeled data, and improves the data scale, accuracy, data generalization and construction efficiency of the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 is a schematic flow chart of a first embodiment of a method for constructing a training data set for training an action recognition model provided by an embodiment of the present application;

[0015] Figure 2 It is a schematic diagram of an action instance generation process of a method for constructing a training data set for training an action recognition model provided by an embodiment of the present application;

[0016] Figure 3 It is a schematic diagram of an initial data set generation process of a method for constructing a training data set for training an action recognition model provided in an embodiment of the present application;

[0017] Figure 4 is a schematic flow chart of a second embodiment of a method for constructing a training data set for training an action recognition model provided by an embodiment of the present application;

[0018] Figure 5 is a schematic flow chart of a third embodiment of a method for constructing a training data set for training an action recognition model provided by an embodiment of the present application;

[0019] Figure 6 The embodiment of the present application provides a schematic diagram of an augmented video generation process for constructing a training data set for training an action recognition model;

[0020] Figure 7 is a schematic block diagram of a device for constructing a training data set for training an action recognition model provided by an embodiment of the present application;

[0021] Figure 8 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0024] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0025] It should be further understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] The embodiment of the present application provides a method for constructing a training data set for training an action recognition model. Among them, the method for constructing a training data set for training an action recognition model can be applied to a server, and a target action event can be generated according to actual needs, an initial data set can be obtained according to the target action event, and the initial data set can be augmented, thereby improving the comprehensiveness of the data set, and no manual annotation of data is required, simplifying the construction process of the data set, avoiding problems such as incorrect annotation and inconsistent annotation caused by manual experience errors, and improving the accuracy and construction efficiency of the data set. Among them, the server can be an independent server or a server cluster.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] See also Figure 1 , Figure 1This is a schematic flow chart of a method for constructing a training data set for training an action recognition model provided by an embodiment of the present application. The method for constructing a training data set for training an action recognition model can be applied to a server, used to generate a large model through actions, generate a target action instance according to actual needs, obtain an initial data set according to the target action instance, and generalize the initial data set with the help of a large visual model to generate an augmented data set as target training data, thereby improving the use value of data in the field of AI and solving the defects of poor generalization, small scale and high generation cost of traditional simulation data; and the generation method does not rely on manually labeled data, avoiding problems such as incorrect labeling and inconsistent labeling caused by manual experience errors, solving the defects of difficult scale, poor accuracy, high cost and low efficiency of manually labeled data, and improving the data scale, accuracy, data generalization and construction efficiency of the data set.

[0029] like Figure 1 As shown, the method for constructing a training data set for training an action recognition model specifically includes steps S101 to S103.

[0030] S101, generating a large model based on a preset action, processing the preset action description information, and generating a target action instance;

[0031] In one embodiment, the action description information is a specific description of the character's action event, and can be freely set by the user according to actual needs.

[0032] In one embodiment, Figure 2 As shown, Figure 2 A schematic diagram of an action instance generation process of a method for constructing a training data set for training an action recognition model provided in an embodiment of the present application. The action generation model can generate corresponding action events according to action description information, and then perform bone redirection on the corresponding action skeleton in the action event to obtain a target action instance.

[0033] Specifically, first, the action description information is processed using the action generation model to generate an initial action event, wherein the human skeleton of the initial action event is in the skeleton format of the SMPL (Skinned Multi-Person Linear Model, parameterized human body model) model. Then, the SMPL skeleton in the initial action event is redirected to the skeleton format of the Mixamo model (it can also be other skeleton models and skeleton formats set by the user according to actual needs) to obtain the target action instance. The Mixamo model is the skeleton format used by Mixamo, an online 3D character animation design and production platform.

[0034] In one embodiment, the motion generation model can be an MDM (Motion Diffusion Model) model, or other models set by the user as needed. The MDM model is a generative machine learning model based on a diffusion model, which is specifically used to generate realistic human motion sequences based on text.

[0035] S102, constructing a scene animation based on the target action instance, and processing the scene animation to obtain target parameter data, and generating an initial data set based on the target parameter data;

[0036] Furthermore, the target parameter data includes video frames, synchronously annotated frame data and camera data. Based on the target action instance, a scene animation is constructed, and the scene animation is processed to obtain target parameter data. Based on the target parameter data, an initial data set is generated, including: transmitting the target action instance to a preset scene, and generating the scene animation based on the motion state of the target action instance and the preset scene; rendering the preset scene and the scene animation based on a preset rendering engine to obtain video frames corresponding to the scene animation, and making the target action instance and the scene data of the preset scene correspond to the video frame to obtain synchronously annotated frame data and camera data with aligned spatiotemporal parameters; obtaining the initial data set based on the video frame, the synchronously annotated frame data and the camera data.

[0037] Furthermore, the synchronously annotated frame data includes action labels, target model bounding boxes, skeletal joint information representing actions, scene depth image data, and scene normal vector image data.

[0038] In one embodiment, an action instance generally refers to an instance in which a specific character model or virtual character performs a specific action in 3D animation, game development, motion capture or other related fields.

[0039] In one embodiment, the preset scene is a scene generated by the UE5 program according to a preset scene model file. The scene model refers to 3D objects that constitute the game world or scene, such as buildings, trees, props, etc. UE5, namely Unreal Engine 5, is a game engine used to create high-end 3D content and interactive experience.

[0040] In a specific embodiment, Figure 3 As shown, Figure 3A schematic diagram of the initial data set generation process of a method for constructing a training data set for training an action recognition model provided by an embodiment of the present application. First, the action model file in the fbx format corresponding to the action instance obtained by redirection and the scene model file preset by the user are imported into the UE5 program, wherein the import step can be executed by an instruction, specifically, after the skeleton redirection completes the acquisition of the file in the fbx format, the import instruction is called to import the action model file and the scene model file into the UE5 program. Then, an animation character class and a character action instance are constructed, and the public parameters of the character action instance are set as a character identifier and a task action list. Then, the character action instance is placed in a scene generated according to the scene model file, wherein the identifier of each character action instance of each frame is unique, and the identifier of the same action instance is consistent throughout the action sequence. Finally, the skeletal action of the target action instance based on the mixamo skeleton is assigned to all the character action instances in the scene with clear logic using the level sequence tool. When the character action changes, the action parameters in its action list are converted by entering the key frame to complete the establishment of the scene animation and obtain the scene animation.

[0041] In one embodiment, the initial data set includes video frames, synchronously annotated frame data, and camera data. The synchronously annotated frame data includes action labels, target model bounding boxes (i.e., Figure 3 The position labels in the ), the skeletal joint information representing the action (i.e. Figure 3 pose information in the scene), scene depth image data (i.e. Figure 3 depth information in ) and scene normal image data (i.e. Figure 3 Specifically, in the process of processing the scene animation by UE5, the video frame corresponding to the scene animation is obtained, and the scene data of the target action instance and the preset scene are matched with the video frame, and the synchronous annotated frame data and the camera data with the time and space parameters aligned are obtained, and the video frame, the synchronous annotated frame data and the camera data are combined to obtain the initial data set.

[0042] In one embodiment, a video frame is composed of a series of static images. In this embodiment, the static images are Figure 3 The camera data includes information such as the camera position, rotation, and focal length for each frame, which can be used to understand the structure and perspective changes of the scene.

[0043] In a specific embodiment, the step of obtaining scene normal vector image data includes: obtaining scene object normal vector values, creating normal vector materials based on the scene object normal vector values, and obtaining scene normal vector image data based on the normal vector material and the scene animation.

[0044] In one embodiment, scene object normal vector data is obtained and mapped to RGB space.

[0045] In a specific embodiment, the scene object normal vector data, i.e., the normal vector values ​​in the three directions of the x-axis, y-axis, and z-axis, are mapped to the G, R, and B channels of the RGB space, and used as the self-luminous color of the material to complete the creation of the normal vector material. Then, a post-processing volume block is added to the scene, its scope is set to infinite, and its post-processing material is set to an instance of the normal vector material, and the scene animation constructed based on the target action instance is re-recorded to obtain the final scene normal vector image data.

[0046] In a specific embodiment, the step of obtaining scene depth image data includes: obtaining a first depth value of a scene animation, obtaining a second depth value based on a second preset formula and the first depth value, creating a depth material based on the second depth value, and obtaining scene depth image data based on the depth material and the scene animation.

[0047] In one embodiment, the first depth value is a scene depth value X recorded in the current scene, and the second depth value is a depth value Y obtained by processing the scene depth value X using a second preset formula.

[0048] In a specific embodiment, a depth image data is obtained by creating a new depth material and processing the depth material. Specifically, the scene depth value X recorded in the current scene is used as input, and the depth value Y is obtained by a second preset formula, wherein the second preset formula is:

[0049]

[0050] After obtaining the depth value Y, the depth change rate is adjusted using the visualization curve function in UE5, and the final depth value is mapped to the RGB space as a grayscale value, that is, the depth value is assigned to the three channels of R, G, and B at the same time to generate a uniform color, which is used as the self-luminous color of the material to complete the creation of the depth material. Then, add a post-processing volume block to the scene, set its scope to infinite and set its post-processing material to an instance of the depth material, and re-record the scene animation built based on the target action instance to obtain the final scene depth image data.

[0051] In a specific embodiment, the step of obtaining the skeletal joint point information includes: skeletally marking the action skeletal key points corresponding to the target action instance to determine multiple skeletal marking points of the target action instance; obtaining the world coordinate values ​​of the skeletal marking points and the posture and viewing frustum information of the rendering camera; obtaining the projection transformation matrix based on the posture and viewing frustum information of the rendering camera; converting the world coordinate values ​​into viewport coordinate values ​​based on the projection transformation matrix to obtain the final skeletal joint point information.

[0052] In one embodiment, skeleton marking refers to marking key points (such as joint points) of each skeleton, so that when the skeleton moves, the action of the character can be accurately tracked and represented.

[0053] In one embodiment, after the skeleton marking is completed, the positions of these skeleton marking points can be determined for each action instance (such as walking, running, etc.), wherein these positions define the posture of the character at a specific point in time.

[0054] In one embodiment, each skeletal marker point has a world coordinate, which indicates the position of the point in the entire scene. The coordinate value of each skeletal marker point in the world coordinate system is obtained.

[0055] In one embodiment, the rendering camera determines the angle and position of the scene. The pose (position and attitude) describes the position and orientation of the camera in the scene. The frustum information defines the range of space that the camera can see, that is, which objects are visible and which are blocked.

[0056] In one embodiment, the projection transformation matrix is ​​the key to converting 3D world coordinates to 2D screen coordinates. Combined with the camera's pose and frustum information, it ensures that only objects within the camera's field of view are rendered correctly.

[0057] In one embodiment, the viewport coordinate values ​​are typically pixel positions on the screen, which determine the final position of the object in the rendered image.

[0058] In one embodiment, the world coordinate values ​​of the skeletal marker points are converted into viewport coordinate values ​​through a projection transformation matrix, and the position data of the human skeletal marker points on the screen, i.e., the human skeletal marker point data, are obtained, which can be used for further rendering, animation processing or analysis.

[0059] In a specific embodiment, 17 slots are established in the Mixamo general skeleton through skeleton editing, corresponding to the 17-point skeleton markers of the OpenPose skeleton, for locating the key points of the skeleton. In the rendering of the scene animation, the world coordinate values ​​of the skeleton markers of each action instance at the same time and the information of the pose and frustum of the rendering camera can be obtained, and the projection transformation matrix is ​​calculated according to the pose and frustum information of the rendering camera, and the coordinates of the 17 OpenPose skeleton nodes are converted into viewport coordinates according to the projection transformation matrix and output, and finally the human skeleton joint point information of each action instance is obtained and output.

[0060] Among them, Mixamo is a platform that provides 3D character animation, and Mixamo universal skeleton is a standard skeleton structure used to create and edit 3D character animation.

[0061] In one embodiment, slot generally refers to a placeholder or slot for placing specific data or components. In the embodiment of the present application, establishing 17 slots means creating 17 specific locations or interfaces on the Mixamo universal skeleton for matching and docking with the 17-point skeleton markers of OpenPose.

[0062] In one embodiment, OpenPose is an open source human pose estimation library that can extract human skeleton point information from images or videos. The 17-point skeleton markers of the openpose skeleton refer to the 17 key skeleton points that OpenPose can identify and locate, usually including the main joints of the human body, such as the head, neck, shoulder, elbow, wrist, hip, knee and ankle.

[0063] In one embodiment, by matching the slots of Mixamo's universal skeleton with the 17-point skeleton markers of OpenPose, the key points of the 3D character skeleton can be accurately located. This helps to accurately apply the human posture information recognized by OpenPose from images or videos to Mixamo's 3D characters, thereby achieving more realistic and natural animation effects.

[0064] In a specific embodiment, the step of obtaining the target model bounding box includes: rendering the scene animation based on preset rendering parameters, and when rendering the basic image data, determining whether there is a character action instance for each pixel. If there is a character action instance, the character action instance and its action type are recorded at this pixel position, and based on the pixel position, the bounding box of the target model of each character action instance is obtained.

[0065] In one embodiment, rendering is the process of converting a three-dimensional scene into a two-dimensional image. In this process, the scene animation needs to be rendered according to preset rendering parameters (such as lighting, material, camera perspective, etc.) to obtain basic image data, usually one or more two-dimensional images, which contain all elements in the scene, including character action instances.

[0066] In one embodiment, when rendering basic image data, it is determined whether there is a character action instance for each pixel, and if there is a character action instance, the character action instance at the pixel position is recorded. At the same time, the action type of the character action instance is also recorded, such as walking, running, jumping, etc.

[0067] In one embodiment, the bounding box of the character action instance may be obtained by calculating the pixel position of the character action instance in the basic image data.

[0068] In a specific embodiment, first, a file with a string of length 11 as the file name is randomly generated, and the name of each frame is set to a string filled with 6-bit frame numbers, and the viewport size (Xv, Yv) of the rendering frame and the rendering frame rate are set to 30FPS (Frames Per Second) and other parameters. After configuring the rendering settings, enter the rendering link to obtain the basic image data and the discriminator information and action list information of the action instance recorded in each frame pixel. After the recording is completed, according to the discriminator information of each action instance, the position (x, y) of the pixel belonging to each action instance is recorded, and the minimum xmin value of the pixel position in the X-axis direction and the maximum xmax value of the pixel position in the X-axis direction are calculated. The minimum ymin value of the pixel position in the Y-axis direction and the maximum ymax value of the pixel position in the Y-axis direction are recorded. The bounding box [xmin, xmax, ymin, ymax] of the current action instance in the current frame is recorded. At the same time, the action type list of the current action instance in the current frame is also recorded. Finally, the bounding box and action type tag data of all action instances in all frames are summarized and output as a txt format file.

[0069] In the above embodiment, the scene animation is rendered by using preset rendering parameters to obtain basic image data, and for each frame pixel, it is determined whether there is a character action instance, and the character action instance and its action type information are recorded; finally, the action instance bounding box is obtained according to the pixel position. There is no need to manually annotate data, which avoids problems such as incorrect annotation and inconsistent annotation caused by manual experience errors, and improves the accuracy of the data set.

[0070] S103, processing the initial data set to obtain model parameters, and generating a large model based on a preset image and the model parameters to generate an augmented data set as a target training data set.

[0071] In one embodiment, the model parameters include a first model parameter and a second model parameter. First, the target parameter data in the initial data set is processed based on the image generation control model to obtain the first model parameter of the image generation model. Then, the preset sample requirements are processed to obtain image generation prompt words, and the target prompt information is comprehensively constructed according to the image generation prompt words, and the target prompt information is used as the second model parameter. Among them, the image generation control model can be a ControlNet model, or it can be other models set by the user according to actual needs.

[0072] In one embodiment, first, an augmented video of a target action instance is generated by using a preset image generation model, a first model parameter, and a second model parameter, and then the augmented video is processed to obtain an augmented data set.

[0073] In a specific embodiment, a large model is generated based on an image, combined with the first model parameters and the second model parameters, various augmentation operations can be performed on the original video frame sequence (i.e., the video frames in the initial data set mentioned in the above embodiment), and the action instance video is regenerated using AIGC (Artificial Intelligence Generated Content) technology, thereby generating a richer and more representative augmented data set. The augmented data set can be used as a target training data set to train a deep learning model to improve the performance and accuracy of the model.

[0074] In the above embodiment, a target action instance can be generated according to actual needs, an initial data set can be obtained according to the target action instance, and the initial data set can be generalized with the help of a large visual model to generate an augmented data set as target training data, thereby improving the use value of data in the AI ​​field and solving the defects of poor generalization, small scale and high generation cost of traditional simulation data; and the generation method does not rely on manually labeled data, avoiding problems such as incorrect labeling and inconsistent labeling due to human experience errors, and solving the defects of manually labeled data that are difficult to scale, poor accuracy, high cost and low efficiency, and improving the data scale, accuracy, data generalization and construction efficiency of the data set.

[0075] See also Figure 4 , Figure 4 It is a schematic flowchart of a method for constructing a training data set for training an action recognition model provided in an embodiment of the present application.

[0076] like Figure 4 As shown, the method for constructing a training data set for training an action recognition model specifically includes steps S201 to S203.

[0077] S201, generating a large model based on the action, processing the action description information, and generating an initial action event;

[0078] S202, performing skeleton redirection based on the action skeleton corresponding to the initial action event to obtain a target action event;

[0079] S203: Based on the target action event, control the movement of the character model to generate the target action instance.

[0080] In one embodiment, bone redirection is performed based on the action bone corresponding to the initial action event to obtain a target action event, including: based on a first preset formula, converting the rotation weight of the original bone into a rotation matrix, and converting the rotation matrix into a quaternion, wherein the original bone is the bone corresponding to the initial action event; redirecting the quaternion corresponding to the original bone to a preset target bone to obtain a target action event.

[0081] In one embodiment, the action description information is a specific description of the initial action event, and can be freely set by the user according to actual needs.

[0082] In a specific embodiment, the action description information is processed using the action generation model to generate an initial action event, wherein the human skeleton of the initial action event can be in the skeleton format of the SMPL (Skinned Multi-Person Linear Model, parameterized human body model) model. Then, the SMPL skeleton in the initial action event is redirected to the skeleton format of the Mixamo model (it can also be other skeleton models and skeleton formats set by the user according to actual needs) to obtain the target action event.

[0083] In a specific embodiment, the action data of the generated initial action event is cleaned and processed to obtain the bone rotation data in the data format of [24, 6, frames] and the root node displacement data in the data format of [3, frames]. Then, the bone rotation data is converted into the axis / angle representation and matched one by one with the 24 nodes of SMPL. Finally, the SMPL format action rotation data in the data format of [24, 3, frames] and the root node displacement data in the data format of [3, frames] are obtained. The rotation weights d6 of all the original bones are converted into the rotation matrix [b1, b2, b3] according to the first preset formula, wherein the first preset formula is:

[0084]

[0085] After calculating each key rotation matrix, convert it into the quaternion mode of [w, x, y, z], and make slight adjustments to the redirection of the quaternion rotation value of each bone to obtain the target action event. According to the target action event, the movement of the human model is controlled. After the bones are bound, the target action instance is generated and saved in fbx format.

[0086] Specifically, the original action stack in the Mixamo model is cleared and the action stack and action layer are reset. Each bone data of each frame of the action data recorded in SMPL format is converted into quaternion, and the quaternion rotation [w1, x1, y1, z1] of the bone Pelvis in SMPL is redirected to the quaternion rotation [w2, x2, y2, z2] of the bone Hips in Mixamo, where w2 = w1, x2 = x1, y2 = y1, z2 = z1. The quaternion rotation [w1, x1, y1, z1] of the bone Spine1 in SMPL is redirected to the quaternion rotation [w2, x2, y2, z2] of the bone Spine in Mixamo, where w2 = w1, x2 = x1, y2 = y1, z2 = z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone Spine2 in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone Spine1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone Spine3 in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone Spine2 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone Neck in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone Neck in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone Head in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone Head in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Collar in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftShoulder in Mixamo, where w2=w1,x2=x1+0.1,y2=y1-0.2,z2=z1+0.4. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Collar in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightShoulder in Mixamo, where w2=w1,x2=x1+0.1,y2=y1+0.2,z2=z1-0.4.Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Shoulder in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftArm in Mixamo, where w2=w1,x2=x1+0.5,y2=y1+0.2,z2=z1+0.2. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Shoulder in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightArm in Mixamo, where w2=w1,x2=x1+0.5,y2=y1-0.3,z2=z1-0.2. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Elbow in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftForeArm in Mixamo, where w2=w1,x2=x1,y2=y1+0.6,z2=z1+0.1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Elbow in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightForeArm in Mixamo, where w2=w1,x2=x1,y2=y1-0.6,z2=z1-0.1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Wrist in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftHand in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Wrist in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightHand in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Hand in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftHandThumb1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Hand in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftHandIndex1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1.Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Hand in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftHandMiddle1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Hand in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftHandRing1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Hand in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftHandPinky1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Hand in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightHandThumb1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Hand in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightHandIndex1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Hand in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightHandMiddle1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Hand in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightHandRing1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Hand in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightHandPinky1 in Mixamo, where w2=w1,x2=x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Hip in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftUpLeg in Mixamo, where w2=w1,x2=-x1,y2=y1,z2=z1.Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Knee in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftLeg in Mixamo, where w2=w1,x2=-x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Ankle in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftFoot in Mixamo, where w2=w1,x2=-x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone L_Foot in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone LeftToe_End in Mixamo, where w2=w1,x2=-x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Hip in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightUpLeg in Mixamo, where w2=w1,x2=-x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Knee in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightLeg in Mixamo, where w2=w1,x2=-x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Ankle in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightFoot in Mixamo, where w2=w1,x2=-x1,y2=y1,z2=z1. Redirect the quaternion rotation [w1,x1,y1,z1] of the bone R_Foot in SMPL to the quaternion rotation [w2,x2,y2,z2] of the bone RightToe_End in Mixamo, where w2=w1,x2=-x1,y2=y1,z2=z1. The action redirection is completed and the result is finally exported to fbx format.

[0087] In the above embodiment, an initial action event is generated according to the action description information, and then the action skeleton corresponding to the initial action event is redirected to generate a target action event. The target action event can be generated according to user needs, thereby improving the practicality and comprehensiveness of the data set.

[0088] See also Figure 5 , Figure 5 It is a schematic flowchart of a method for constructing a training data set for training an action recognition model provided in an embodiment of the present application.

[0089] like Figure 5 As shown, the method for constructing a training data set for training an action recognition model specifically includes steps S301 to S303.

[0090] S301, processing the target parameter data in the initial data set based on the image generation control model to obtain the first model parameter of the image generation large model;

[0091] Furthermore, the target parameter data in the initial data set is processed based on the image generation control model to obtain the first model parameters of the image generation large model, including: based on the first sub-model of the image generation control model, the scene depth image data in the initial data set is processed to obtain depth control parameters; based on the second sub-model of the image generation control model, the skeletal joint point information in the initial data set is processed to obtain posture control parameters; based on a preset line draft extraction processor, the video frames in the initial data set are processed to obtain line draft images, and based on the third sub-model of the image generation control model, the line draft control parameters are extracted from the line draft images; based on the fourth sub-model of the image generation control model, the normal vector control parameters are extracted from the scene normal vector image data in the initial data set; the depth control parameters, the posture control parameters, the line draft control parameters and the normal vector control parameters are used as the first model parameters.

[0092] In this embodiment, the image generation control model can be a ControlNet model, which includes a first sub-model: controlnet_Depth model, a second sub-model: controlnet_0penpose model, a third sub-model: controlnet_Lineart model and a fourth sub-model: controlnet_Normal model; it can also be an image generation control model set by other users according to actual needs, which is used to extract depth control parameters from scene depth image data, extract posture control parameters from skeletal joint point information, extract line draft control parameters from line draft images, and extract normal vector control parameters from scene normal vector image data.

[0093] In one embodiment, the skeleton joint point information can be used to track and identify the character action instances in the scene. By combining these marker points with the basic image data, a posture image of the human skeleton can be generated, and then the posture control parameters can be obtained.

[0094] In a specific embodiment, the ControlNet_Depth model is used to process scene depth image data to extract depth control parameters. The human skeleton joint information is used to track and identify character action instances in the scene, generate openpose posture images, and then the ControlNet_OpenPose model is used to extract posture control parameters from the openpose posture images. The images in the original video are processed using a realistic Lineart processor to generate line draft images, and then the ControlNet_Lineart model is used to extract line draft control parameters from the line draft images. The ControlNet_Normal model is used to extract normal vector control parameters from the scene normal vector image data.

[0095] S302, based on the preset sample requirements, obtaining image generation prompt words, processing the image generation prompt words, obtaining target prompt information, and using the target prompt information as the second model parameter of the image generation large model;

[0096] Furthermore, obtaining image generation prompt words based on preset sample requirements includes: performing semantic analysis on the sample requirements to obtain at least one key information corresponding to the sample requirements; constructing at least one image generation prompt word based on each of the key information; wherein the image generation prompt words include reverse inference prompt words, style prompt words and custom prompt words.

[0097] In one embodiment, the sample requirement may be a custom text obtained according to user input content, and may include description or annotation information of the target training data set, such as annotations of scenes, actions, or targets.

[0098] In one embodiment, the reverse prompt words are used to guide the model to reverse the input conditions from the output. The style prompt words are used to specify the style or tone of the image. The custom prompt words are customized according to specific needs. The prompt words generated by each image are merged to obtain the target prompt information.

[0099] S303: Generate a large model based on the image, the first model parameters, and the second model parameters, generate the augmented data set, and use the augmented data set as the target training data set.

[0100] Furthermore, generating a large model based on the image, processing the first model parameters and the second model parameters to generate the augmented data set includes: generating an augmented video of the target action instance based on the large model generated by the image, the first model parameters and the second model parameters; processing the augmented video to obtain the augmented data set.

[0101] In one embodiment, various augmentation operations can be performed on the original video frame sequence by comprehensively using information such as depth control parameters, posture control parameters, normal vector control parameters, line draft control parameters, and augmentation control parameters, and action instance video can be regenerated using AIGC (Artificial Intelligence Generated Content) technology, thereby generating a richer and more representative augmented data set. The augmented data set can be used as a target training data set to train a deep learning model to improve the performance and accuracy of the model.

[0102] In a specific embodiment, Figure 6 As stated, Figure 6 The embodiment of the present application provides a schematic diagram of an augmented video generation process for constructing a training data set for training an action recognition model. Figure 6 depth data in the image), skeleton key point information representing the action (i.e. Figure 6 pose estimation data in ), scene normal image data (i.e. Figure 6 ) and the line drawing data of the original video (i.e. Figure 6 The lineart data in the image are processed to obtain the first model parameters. The inverse prompt words, style prompt words and custom prompt words are processed to obtain the final target prompt information as the second model parameters. IPAdapter, text-to-image diffusion model (such as SD1.5 basic model, DreamShaper model, etc.) and Animatediff model are loaded to generate videos. During the video generation process, the generated video is precisely controlled according to the first model parameters (depth control parameters, posture control parameters, normal vector control parameters, line draft control parameters) and the second model parameters (augmentation control parameters). VAE (Variational AutoEncoder) is used to convert the original video frame sequence containing action instances into a low-dimensional abstract representation and input it into the Ksampler module to generate augmented videos. Finally, the SEGSDetailer method is used to optimize the video (such as hand and face repair) to complete the generation of augmented videos of action event instances. Finally, the generated video is divided into images and saved in jpg format, and the expansion of the data set is finally completed. The extended data set is obtained and used as the target training data set.

[0103] Among them, the ControlNet_Depth model in the Controlnet model is a deep learning model that is specifically designed to use depth image data to influence the image generation model. It can make full use of the information of the depth image to control the video and image generation results. The ControlNet_OpenPose model in the Controlnet model is a deep learning model that is specifically designed to use human posture estimation data to influence the image generation model. It can use the key points and posture data of the human body in the image to control the video and image generation results. The ControlNet_Lineart model in the Controlnet model is a deep learning model that is specifically designed to use image line draft data to influence the image generation model. It can use the line draft or sketch data of the image to control the video and image generation results. The ControlNet_Normal model in the Controlnet model is a deep learning model that is specifically designed to use image normal vector data to influence the image generation model. It can use the normal vector data of the object in the image to control the video and image generation results. IPAdapter is a deep learning model used to add additional information to influence the graphics output model. It enables the pre-trained text-to-image diffusion model to generate images through image prompts, thereby generating more controlled and accurate outputs. The SD1.5 model, or Stable Diffusion 1.5, is a text-to-image diffusion model. The SD1.5 base model and the Animatediff model are commonly used for video generation tasks, where the SD1.5 base model is a versatile text-to-image diffusion model used to generate the basic structure and content of a video, while the SEGSDetailer method is used to refine and optimize details. The Ksampler module is a tool for advanced sampling operations, which is usually used to control the sampling strategies and methods in the image generation process. It allows users to customize the sampling process through various parameters, and generate new data samples by manipulating the representation of the latent space, adjusting conditions, and adjusting the noise level.

[0104] In the above embodiment, the initial data set is generalized with the help of a large visual model to achieve diversified augmentation, and an augmented data set is generated as target training data, which solves the defect of poor generalization of traditional simulation data and improves the generalization ability and effectiveness of the data set; and the method does not rely on manually labeled data, avoiding problems such as incorrect labeling and inconsistent labeling due to human experience errors, and solves the defects of small scale, poor accuracy, high cost and low efficiency of manually labeled data, and improves the data scale, accuracy, data generalization and construction efficiency of the data set.

[0105] See also Figure 7 , Figure 7The embodiment of the present application provides a schematic block diagram of a device for constructing a training data set for training an action recognition model, and the device for constructing a training data set for training an action recognition model is used to execute the aforementioned method for constructing a training data set for training an action recognition model. The device for constructing a training data set for training an action recognition model can be configured on a server.

[0106] like Figure 7 As shown, the device 400 for constructing a training data set for training an action recognition model includes:

[0107] The target action instance generation module 401 is used to generate a large model based on a preset action, process the preset action description information, and generate a target action instance;

[0108] An initial data set generation module 402 is used to construct a scene animation based on the target action instance, process the scene animation, obtain target parameter data, and generate an initial data set based on the target parameter data;

[0109] The target training data set generation module 403 is used to process the initial data set to obtain model parameters, and generate a large model and the model parameters based on a preset image to generate an augmented data set as the target training data set.

[0110] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0111] The above-mentioned device can be implemented in the form of a computer program. Figure 8 Runs on the computer device shown.

[0112] See also Figure 8 , Figure 8 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0113] See also Figure 8 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0114] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for constructing a training data set for training an action recognition model.

[0115] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0116] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any method for constructing a training data set for training an action recognition model.

[0117] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0119] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0120] Generate a large model based on the preset action, process the preset action description information, and generate a target action instance;

[0121] Based on the target action instance, a scene animation is constructed, and the scene animation is processed to obtain target parameter data, and based on the target parameter data, an initial data set is generated;

[0122] The initial data set is processed to obtain model parameters, and a large model and the model parameters are generated based on a preset image to generate an augmented data set as a target training data set.

[0123] In one embodiment, when the processor generates a large model based on a preset action, processes the preset action description information, and generates a target action instance, it is used to implement:

[0124] Based on the action generation macro model, the action description information is processed to generate an initial action event;

[0125] Performing skeleton redirection based on the action skeleton corresponding to the initial action event to obtain a target action event;

[0126] Based on the target action event, the character model movement is controlled to generate the target action instance.

[0127] In one embodiment, the target parameter data includes video frames, synchronously annotated frame data, and camera data. When the processor constructs a scene animation based on the target action instance, processes the scene animation, obtains the target parameter data, and generates an initial data set based on the target parameter data, it is used to implement:

[0128] Transmitting the target action instance to a preset scene, and generating the scene animation based on the motion state of the target action instance and the preset scene;

[0129] Rendering the preset scene and scene animation based on a preset rendering engine to obtain a video frame corresponding to the scene animation, and making the target action instance and the scene data of the preset scene correspond to the video frame to obtain synchronously annotated frame data and camera data aligned with time and space parameters;

[0130] The initial data set is obtained based on the video frame, the synchronously annotated frame data and the camera data.

[0131] In one embodiment, the synchronously annotated frame data includes an action label, a target model bounding box, skeletal joint information representing the action, scene depth image data, and scene normal vector image data.

[0132] In one embodiment, the processor processes the initial data set to obtain model parameters, generates a large model based on a preset image and the model parameters, and generates an augmented data set as a target training data set, and is used to implement:

[0133] Processing the target parameter data in the initial data set based on the image generation control model to obtain the first model parameters of the image generation large model;

[0134] Based on the preset sample requirements, image generation prompt words are obtained, and the image generation prompt words are processed to obtain target prompt information, and the target prompt information is used as the second model parameter of the image generation large model;

[0135] A large model is generated based on the image, the first model parameters, and the second model parameters, and the augmented data set is generated, and the augmented data set is used as the target training data set.

[0136] In one embodiment, when the processor processes the target parameter data in the initial data set based on the image generation control model to obtain the first model parameter of the image generation large model, it is used to implement:

[0137] Based on the first sub-model of the image generation control model, the scene depth image data in the initial data set is processed to obtain a depth control parameter;

[0138] Based on the image generating control model, a second sub-model is used to process the skeletal joint information in the initial data set to obtain posture control parameters;

[0139] Based on a preset line draft extraction processor, the video frames in the initial data set are processed to obtain a line draft image, and based on the image, a third sub-model of the control model is generated to extract line draft control parameters from the line draft image;

[0140] extracting normal vector control parameters from scene normal vector image data in the initial data set based on a fourth sub-model of the image generation control model;

[0141] The depth control parameter, the posture control parameter, the line draft control parameter and the normal vector control parameter are used as the first model parameters.

[0142] In one embodiment, when the processor generates a large model based on the image and processes the first model parameter and the second model parameter to generate the augmented data set, the processor is used to implement:

[0143] Generate an augmented video of the target action instance based on the image generation model, the first model parameters, and the second model parameters;

[0144] The augmented video is processed to obtain the augmented data set.

[0145] In one embodiment, when obtaining the image generation prompt word based on the preset sample requirement, the processor is used to implement:

[0146] Performing semantic analysis on the sample requirement to obtain at least one key information corresponding to the sample requirement;

[0147] Based on each of the key information, construct at least one of the image generation prompt words;

[0148] Performing natural language processing on the prompt words generated by each image to obtain the target prompt information;

[0149] The image generation prompt words include reverse prompt words, style prompt words and custom prompt words.

[0150] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any method provided in an embodiment of the present application for constructing a training data set for training an action recognition model.

[0151] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0152] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for constructing a training data set for training an action recognition model, characterized in that: include: Generate a large model based on the preset action, process the preset action description information, and generate a target action instance; Based on the target action instance, a scene animation is constructed, and the scene animation is processed to obtain target parameter data, and based on the target parameter data, an initial data set is generated; Processing the initial data set to obtain model parameters, and generating a large model based on a preset image and the model parameters to generate an augmented data set as a target training data set; The step of generating a large model based on a preset action, processing the preset action description information, and generating a target action instance includes: Based on the action generation macro model, the action description information is processed to generate an initial action event; Performing skeleton redirection based on the action skeleton corresponding to the initial action event to obtain a target action event; Based on the target action event, controlling the movement of the character model to generate the target action instance; The processing of the initial data set to obtain model parameters, and generating a large model based on a preset image and the model parameters to generate an augmented data set as a target training data set includes: Processing the target parameter data in the initial data set based on the image generation control model to obtain the first model parameters of the image generation large model; Based on the preset sample requirements, image generation prompt words are obtained, and the image generation prompt words are processed to obtain target prompt information, and the target prompt information is used as the second model parameter of the image generation large model; Generate the augmented data set based on the image generation model, the first model parameters and the second model parameters, and use the augmented data set as the target training data set; The step of processing the target parameter data in the initial data set based on the image generation control model to obtain the first model parameter of the image generation large model includes: Based on the first sub-model of the image generation control model, the scene depth image data in the initial data set is processed to obtain a depth control parameter; Based on the image generating control model, a second sub-model is used to process the skeletal joint information in the initial data set to obtain posture control parameters; Based on a preset line draft extraction processor, the video frames in the initial data set are processed to obtain a line draft image, and based on the image, a third sub-model of the control model is generated to extract line draft control parameters from the line draft image; extracting normal vector control parameters from scene normal vector image data in the initial data set based on a fourth sub-model of the image generation control model; The depth control parameter, the posture control parameter, the line draft control parameter and the normal vector control parameter are used as the first model parameters.

2. The method for constructing a training data set for training an action recognition model according to claim 1, characterized in that: The target parameter data includes video frames, synchronously annotated frame data and camera data. Based on the target action instance, a scene animation is constructed, and the scene animation is processed to obtain the target parameter data. Based on the target parameter data, an initial data set is generated, including: Transmitting the target action instance to a preset scene, and generating the scene animation based on the motion state of the target action instance and the preset scene; Rendering the preset scene and scene animation based on a preset rendering engine to obtain a video frame corresponding to the scene animation, and making the target action instance and the scene data of the preset scene correspond to the video frame to obtain synchronously annotated frame data and camera data aligned with time and space parameters; The initial data set is obtained based on the video frame, the synchronously annotated frame data and the camera data.

3. The method for constructing a training data set for training an action recognition model according to claim 2, characterized in that: The synchronously annotated frame data includes an action label, a target model bounding box, skeletal joint point information representing the action, scene depth image data, and scene normal vector image data.

4. The method for constructing a training data set for training an action recognition model according to claim 1, characterized in that: The generating a large model based on the image, processing the first model parameter and the second model parameter to generate the augmented data set includes: Generate an augmented video of the target action instance based on the image generation model, the first model parameters, and the second model parameters; The augmented video is processed to obtain the augmented data set.

5. The method for constructing a training data set for training an action recognition model according to claim 1, characterized in that: The step of obtaining the image generation prompt words based on the preset sample requirements includes: Performing semantic analysis on the sample requirement to obtain at least one key information corresponding to the sample requirement; Based on each of the key information, construct at least one of the image generation prompt words; The image generation prompt words include reverse prompt words, style prompt words and custom prompt words.

6. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the method for constructing a training data set for training an action recognition model as described in any one of claims 1 to 5 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the method for constructing a training data set for training an action recognition model as described in any one of claims 1 to 5.

Images