Behavior marking method and device of target object, electronic equipment and medium

Through joint training based on key points and preset models in the prior art, the accuracy of target object behavior labeling is improved, the problem of inaccurate behavior labeling in the prior art is solved, and more efficient behavior recognition and labeling is achieved.

CN119091508AInactive Publication Date: 2024-12-06ANHUI ZHUOZHI EDUCATION TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411222485.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the prior art, the behavior labeling of target objects is not very accurate, and it is difficult to effectively identify and classify target objects' behavior in images.

Method used

By obtaining the initial behavior labeling results based on the key points of the image to be identified, training the candidate behavior labeling model with the preset model and the behavior labeling results until the model converges, obtaining the target behavior labeling model, and using this model for behavior labeling.

Benefits of technology

The accuracy of target object behavior annotation is improved, and the reliability of behavior annotation results is enhanced through various methods of joint annotation and model training.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091508A_ABST
    Figure CN119091508A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a behavior labeling method and device for a target object, electronic equipment and a medium, and the method comprises the steps: obtaining a first behavior labeling result of the target object in a first to-be-recognized picture based on a key point in the first to-be-recognized picture; based on a first preset model and the first behavior labeling result, obtaining a second behavior labeling result of a target object in the first to-be-recognized picture; based on the first to-be-recognized picture, the second behavior labeling result and the third behavior labeling result, training a candidate behavior labeling model until the candidate behavior labeling model converges, and obtaining a target behavior labeling model; and performing behavior labeling of the target object based on the target behavior labeling model. According to the embodiment of the invention, the candidate behavior labeling model is trained by using the behavior labeling result obtained by performing joint labeling in multiple modes, the target behavior labeling model is obtained, behavior labeling is performed on the target object based on the target behavior labeling model, and the accuracy of behavior labeling of the target object is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of computer vision, and specifically relates to a method, device, electronic device and medium for labeling the behavior of a target object. Background Art

[0002] In some scenarios, automatically detecting and analyzing the behaviors of target objects (e.g., using electronic devices, raising hands, reading, etc.) (for example, in educational scenarios, the target objects can be students) is crucial to collecting feedback on engagement and attention, which helps relevant personnel optimize interaction strategies and improve the overall quality of interaction.

[0003] In the prior art, behavior recognition technology generally uses instance segmentation models to identify and classify the behavior of target objects in images. First, based on target detection, the target object in the image is determined, and then the feature representation of the target object is extracted through the Feature Pyramid Network (FCN). Subsequently, pixel-level classification is performed, and finally a mask is generated, which depicts the shape and outline of the target object in the image. The behavior of the target object is annotated based on the shape and outline of the target object in the image.

[0004] However, using the existing technology, the accuracy of the target object's behavior labeling is not high. Summary of the invention

[0005] Embodiments of the present application provide a method, device, electronic device, and medium for labeling a behavior of a target object, so as to solve the problem of low reliability of the behavior labeling of the target object in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a method for labeling a behavior of a target object, including:

[0007] Based on the key points in the first picture to be identified, obtaining a labeling result for a first line of the target object in the first picture to be identified;

[0008] Based on the first preset model and the first line annotation result, obtaining a second line annotation result of the target object in the first to-be-identified picture;

[0009] Based on the first image to be identified, the second behavior annotation result, and the third behavior annotation result, training a candidate behavior annotation model until the candidate behavior annotation model converges to obtain a target behavior annotation model, wherein the third behavior annotation result is a behavior annotation result obtained by annotating the first image to be identified based on the candidate behavior annotation model;

[0010] The target object's behavior is labeled based on the target behavior labeling model.

[0011] Optionally, the training of a candidate behavior annotation model based on the first to-be-recognized image, the second behavior annotation result, and the third behavior annotation result until the candidate behavior annotation model converges to obtain a target behavior annotation model includes:

[0012] If the second behavior labeling result is consistent with the third behavior labeling result, the second behavior labeling result or the third behavior labeling result is used as the target behavior labeling result, and based on the target behavior labeling result, the candidate behavior labeling model is trained until the candidate behavior labeling model converges to obtain the target behavior labeling model;

[0013] If the second behavior labeling result is inconsistent with the third behavior labeling result, the manual labeling result is used as the target behavior labeling result, and the candidate behavior labeling model is trained based on the target behavior labeling result until the candidate behavior labeling model converges to obtain the target behavior labeling model.

[0014] Optionally, obtaining a second behavior of the target object in the first to-be-identified picture as a labeling result based on the first preset model and the first behavior as a labeling result includes:

[0015] Inputting the first image to be identified into a first preset model to obtain a second candidate annotation result;

[0016] Based on the consistency between the first line annotation result and the second candidate annotation result, the second line annotation result is obtained.

[0017] Optionally, obtaining the second line annotation result based on the consistency between the first line annotation result and the second candidate annotation result includes:

[0018] If the first behavior annotation result and the second candidate annotation result are consistent, determining that the first behavior is the annotation result or the second candidate annotation result is the second behavior annotation result;

[0019] If the first line annotation result is inconsistent with the second candidate annotation result, a manual annotation result is obtained, and the manual annotation result is determined to be the second line annotation result.

[0020] Optionally, the step of inputting the first image to be identified into a first preset model to obtain a second candidate annotation result includes:

[0021] Based on inputting the first picture to be recognized into a visual question answering model;

[0022] Obtaining a second initial annotation result obtained in the classification prompt step of the visual question answering model;

[0023] If the confidence of the second initial annotation result is greater than or equal to a preset threshold in the classification confirmation step of the visual question answering model, determining the second initial annotation result as a second candidate annotation result;

[0024] If the confidence of the second initial annotation result is less than the preset threshold in the classification confirmation step of the visual question answering model, a manual annotation result is obtained and the manual annotation result is determined to be a second candidate annotation result.

[0025] Optionally, obtaining the first line of the target object in the first picture to be identified as a labeling result based on the key points in the first picture to be identified includes:

[0026] Based on the human body detection model, obtaining an image of the target object in the first picture to be identified;

[0027] Based on a human posture recognition model, obtaining key point coordinates in the image of the target object;

[0028] Based on the key point coordinates, obtaining the head height and head width of the target object;

[0029] Based on the key point coordinates, the head height and the head width, a first line is obtained as a labeling result.

[0030] Optionally, the key point coordinates include: facial key point coordinates and body key point coordinates;

[0031] The step of obtaining the head height and the head width based on the key point coordinates includes:

[0032] Based on the coordinates of facial key points, obtaining the head height;

[0033] Based on the coordinates of the key points of the body, the head width is obtained.

[0034] Optionally, the facial key point coordinates include:

[0035] nose coordinates, first-side eye coordinates, second-side eye coordinates, first-side ear coordinates, and second-side ear coordinates;

[0036] The step of obtaining the head height based on the facial key point coordinates includes:

[0037] The head height is obtained based on an average value of the ordinate of the nose coordinate, the ordinate of the first eye coordinate, the ordinate of the second eye coordinate, the ordinate of the first ear coordinate, and the ordinate of the second ear coordinate.

[0038] Optionally, the body key point coordinates include: first side shoulder coordinates and second side shoulder coordinates;

[0039] The step of obtaining the head width based on the coordinates of key points of the body includes:

[0040] The head width is obtained based on half of the difference between the abscissa of the first shoulder coordinate and the abscissa of the second shoulder coordinate.

[0041] Optionally, the body key point coordinates further include: first side elbow joint coordinates, second side elbow joint coordinates, first side wrist joint coordinates, and second side wrist joint coordinates;

[0042] The obtaining the first line as a labeling result based on the key point coordinates, the head height and the head width includes:

[0043] If the ordinate of the first wrist joint coordinate plus half of the head width is less than the ordinate of the first elbow joint coordinate; and the ordinate of the first elbow joint coordinate plus half of the head width is less than the ordinate of the first shoulder coordinate; and the ordinate of the first wrist joint coordinate plus the head width is less than the head height, then it is determined that the first line is annotated as the target object being in the first posture;

[0044] or,

[0045] If the ordinate of the second wrist joint coordinate plus half of the head width is less than the ordinate of the second elbow joint coordinate; and the ordinate of the second elbow joint coordinate plus half of the head width is less than the ordinate of the second shoulder coordinate; and the ordinate of the second wrist joint coordinate plus the head width is less than the head height, then it is determined that the first line is annotated as the target object being in the first posture;

[0046] Optionally, the method further includes:

[0047] If the ordinate of the wrist joint coordinate on the first side plus half the head width is less than the ordinate of the elbow joint coordinate on the first side; and the ordinate of the elbow joint coordinate on the first side plus half the head width is less than the ordinate of the shoulder coordinate on the first side; and the ordinate of the wrist joint coordinate on the first side plus half the head width is less than the head height; the ordinate of the wrist joint coordinate on the second side plus half the head width is less than the ordinate of the elbow joint coordinate on the second side; and the ordinate of the elbow joint coordinate on the second side plus half the head width is less than the ordinate of the shoulder coordinate on the second side; and the ordinate of the wrist joint coordinate on the second side plus half the head width is less than the head height, then it is determined that the first behavior is annotated as the target object being in the second posture.

[0048] Optionally, the body key point coordinates further include: first side hip coordinates and second side hip coordinates;

[0049] The method further comprises:

[0050] If at least four of the following conditions are met, determining that the first behavior is labeled as the target object being in a third posture;

[0051] The abscissa of the eye coordinate on the first side is less than the abscissa of the eye coordinate on the second side;

[0052] The abscissa of the first side ear coordinate is smaller than the abscissa of the second side ear coordinate;

[0053] The abscissa of the first side shoulder coordinate is smaller than the abscissa of the second side shoulder coordinate;

[0054] The abscissa of the first side elbow joint coordinate is smaller than the abscissa of the second side elbow joint coordinate;

[0055] The abscissa of the first-side wrist joint coordinate is smaller than the abscissa of the second-side wrist joint coordinate;

[0056] The abscissa of the first side hip coordinate is smaller than the abscissa of the second side hip coordinate.

[0057] In a second aspect, an embodiment of the present application provides a behavior labeling device for a target object, including:

[0058] A first processing module, configured to obtain a labeling result of a first line of a target object in a first picture to be identified based on key points in the first picture to be identified;

[0059] A second processing module, configured to obtain a second line annotation result of the target object in the first to-be-identified picture based on the first preset model and the first line annotation result;

[0060] a third processing module, configured to train a candidate behavior annotation model based on the first image to be identified, the second behavior annotation result, and the third behavior annotation result, until the candidate behavior annotation model converges to obtain a target behavior annotation model, wherein the third behavior annotation result is a behavior annotation result obtained by annotating the first image to be identified based on the candidate behavior annotation model;

[0061] The labeling module is used to label the behavior of the target object based on the target behavior labeling model.

[0062] Optionally, the first processing module is further used to obtain an image of a target object in the first picture to be identified based on a human body detection model;

[0063] Based on a human posture recognition model, obtaining key point coordinates in the image of the target object;

[0064] Based on the key point coordinates, obtaining the head height and head width of the target object;

[0065] Based on the key point coordinates, the head height and the head width, a first line is obtained as a labeling result.

[0066] Optionally, the first processing module is specifically used to obtain key point coordinates, including: facial key point coordinates and body key point coordinates;

[0067] The step of obtaining the head height and the head width based on the key point coordinates includes:

[0068] Based on the coordinates of facial key points, obtaining the head height;

[0069] Based on the coordinates of the key points of the body, the head width is obtained.

[0070] Optionally, the facial key point coordinates include:

[0071] nose coordinates, first-side eye coordinates, second-side eye coordinates, first-side ear coordinates, and second-side ear coordinates;

[0072] The step of obtaining the head height based on the facial key point coordinates includes:

[0073] The head height is obtained based on an average value of the ordinate of the nose coordinate, the ordinate of the first eye coordinate, the ordinate of the second eye coordinate, the ordinate of the first ear coordinate, and the ordinate of the second ear coordinate.

[0074] Optionally, the body key point coordinates include: first side shoulder coordinates and second side shoulder coordinates;

[0075] The step of obtaining the head width based on the coordinates of key points of the body includes:

[0076] The head width is obtained based on half of the difference between the abscissa of the first shoulder coordinate and the abscissa of the second shoulder coordinate.

[0077] Optionally, the body key point coordinates further include: first side elbow joint coordinates, second side elbow joint coordinates, first side wrist joint coordinates, and second side wrist joint coordinates;

[0078] The obtaining the first line as a labeling result based on the key point coordinates, the head height and the head width includes:

[0079] If the ordinate of the first wrist joint coordinate plus half of the head width is less than the ordinate of the first elbow joint coordinate; and the ordinate of the first elbow joint coordinate plus half of the head width is less than the ordinate of the first shoulder coordinate; and the ordinate of the first wrist joint coordinate plus the head width is less than the head height, then it is determined that the first line is annotated as the target object being in the first posture;

[0080] or,

[0081] If the ordinate of the second wrist joint coordinate plus half of the head width is less than the ordinate of the second elbow joint coordinate; and the ordinate of the second elbow joint coordinate plus half of the head width is less than the ordinate of the second shoulder coordinate; and the ordinate of the second wrist joint coordinate plus the head width is less than the head height, then it is determined that the first line is annotated as the target object being in the first posture;

[0082] Optionally, the method further includes:

[0083] If the ordinate of the wrist joint coordinate on the first side plus half the head width is less than the ordinate of the elbow joint coordinate on the first side; and the ordinate of the elbow joint coordinate on the first side plus half the head width is less than the ordinate of the shoulder coordinate on the first side; and the ordinate of the wrist joint coordinate on the first side plus half the head width is less than the head height; the ordinate of the wrist joint coordinate on the second side plus half the head width is less than the ordinate of the elbow joint coordinate on the second side; and the ordinate of the elbow joint coordinate on the second side plus half the head width is less than the ordinate of the shoulder coordinate on the second side; and the ordinate of the wrist joint coordinate on the second side plus half the head width is less than the head height, then it is determined that the first behavior is annotated as the target object being in the second posture.

[0084] Optionally, the body key point coordinates further include: first side hip coordinates and second side hip coordinates;

[0085] The method further comprises:

[0086] If at least four of the following conditions are met, determining that the first behavior is labeled as the target object being in a third posture;

[0087] The abscissa of the eye coordinate on the first side is less than the abscissa of the eye coordinate on the second side;

[0088] The abscissa of the first side ear coordinate is smaller than the abscissa of the second side ear coordinate;

[0089] The abscissa of the first side shoulder coordinate is smaller than the abscissa of the second side shoulder coordinate;

[0090] The abscissa of the first side elbow joint coordinate is smaller than the abscissa of the second side elbow joint coordinate;

[0091] The abscissa of the first-side wrist joint coordinate is smaller than the abscissa of the second-side wrist joint coordinate;

[0092] The abscissa of the first side hip coordinate is smaller than the abscissa of the second side hip coordinate.

[0093] Optionally, the second processing module is specifically configured to obtain, based on the first preset model and the first behavior annotation result, a second behavior annotation result of the target object in the first to-be-identified picture, including:

[0094] Inputting the first image to be identified into a first preset model to obtain a second candidate annotation result;

[0095] Based on the consistency between the first line annotation result and the second candidate annotation result, the second line annotation result is obtained.

[0096] Optionally, the second processing module is further configured to input the first image to be identified into a visual question answering model;

[0097] Obtaining a second initial annotation result obtained in the classification prompt step of the visual question answering model;

[0098] If the confidence of the second initial annotation result is greater than or equal to a preset threshold in the classification confirmation step of the visual question answering model, determining the second initial annotation result as a second candidate annotation result;

[0099] If the confidence of the second initial annotation result is less than the preset threshold in the classification confirmation step of the visual question answering model, a manual annotation result is obtained and the manual annotation result is determined to be a second candidate annotation result.

[0100] Optionally, the second processing module is further configured to obtain the second line annotation result based on the consistency between the first line annotation result and the second candidate annotation result, including:

[0101] If the first behavior annotation result and the second candidate annotation result are consistent, determining that the first behavior is the annotation result or the second candidate annotation result is the second behavior annotation result;

[0102] If the first line annotation result is inconsistent with the second candidate annotation result, a manual annotation result is obtained, and the manual annotation result is determined to be the second line annotation result.

[0103] Optionally, the third processing module is specifically used to train a candidate behavior annotation model based on the first to-be-recognized image, the second behavior annotation result, and the third behavior annotation result until the candidate behavior annotation model converges to obtain a target behavior annotation model, including:

[0104] If the second behavior labeling result is consistent with the third behavior labeling result, the second behavior labeling result or the third behavior labeling result is used as the target behavior labeling result, and based on the target behavior labeling result, the candidate behavior labeling model is trained until the candidate behavior labeling model converges to obtain the target behavior labeling model;

[0105] If the second behavior labeling result is inconsistent with the third behavior labeling result, the manual labeling result is used as the target behavior labeling result, and the candidate behavior labeling model is trained based on the target behavior labeling result until the candidate behavior labeling model converges to obtain the target behavior labeling model.

[0106] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the target object behavior labeling method as described in any one of the first aspects are implemented.

[0107] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the target object behavior labeling method as described in any one of the first aspects are implemented.

[0108] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed by a processor of a vehicle or a cloud server, implements the steps of the target object behavior labeling method as described in any one of the first aspects.

[0109] The target object behavior labeling method, device, electronic device and medium provided in the embodiments of the present application obtain the first behavior labeling result of the target object in the first picture to be identified based on the key points in the first picture to be identified; obtain the second behavior labeling result of the target object in the first picture to be identified based on the first preset model and the first behavior labeling result; train the candidate behavior labeling model based on the first picture to be identified, the second behavior labeling result and the third behavior labeling result until the candidate behavior labeling model converges to obtain the target behavior labeling model; and label the behavior of the target object based on the target behavior labeling model. The embodiments of the present application train the candidate behavior labeling model with the behavior labeling results obtained by joint labeling in multiple ways to obtain the target behavior labeling model, and label the behavior of the target object based on the target behavior labeling model, thereby improving the accuracy of the target object behavior labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 A flowchart of a method for labeling a target object's behavior provided in an embodiment of the present application;

[0111] Figure 2 A flowchart of another method for labeling target object behaviors provided in an embodiment of the present application;

[0112] Figure 3 A schematic diagram of a coordinate axis provided in an embodiment of the present application;

[0113] Figure 4 A flowchart of another method for labeling behavior of a target object provided in an embodiment of the present application;

[0114] Figure 5 A flowchart of another method for labeling behavior of a target object provided in an embodiment of the present application;

[0115] Figure 6 A flowchart of another method for labeling behavior of a target object provided in an embodiment of the present application;

[0116] Figure 7 A flowchart of another method for labeling behavior of a target object provided in an embodiment of the present application;

[0117] Figure 8 A schematic diagram of the structure of a target object behavior labeling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0118] The technical solutions in the embodiments of the present application will be clearly described below in conjunction with the drawings in the embodiments of the present application. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of the present application.

[0119] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited.

[0120] In the related art, the accuracy of target object behavior labeling is not high due to the fact that the target object may be blocked, the lighting conditions, the background interference, etc.

[0121] In order to improve the accuracy of behavior labeling, the present application provides a behavior labeling method for a target object. The method can train a candidate behavior labeling model based on the behavior labeling results obtained by jointly labeling a first picture to be identified, a first behavior labeling result, a second behavior labeling result, and a third behavior labeling result in multiple ways until the candidate behavior labeling model converges to obtain a target behavior labeling model, and then label the behavior of the target object based on the target behavior labeling model.

[0122] In each embodiment of the present application, the behavior annotation of the target object is performed based on the authorization of the target object. The technical solution of the present application is described below with several specific embodiments.

[0123] Figure 1 A flow chart of a method for behavior labeling of a target object provided in an embodiment of the present application. This embodiment is performed by a target object behavior labeling device, which may be jointly composed of a single-stage behavior detection model and a visual question-answering model. The second behavior labeling result is obtained by jointly comparing the first behavior labeling result determined by the key points detected by the behavior detection model with the second candidate labeling result obtained by the visual question-answering model, and the first image to be identified is labeled based on the candidate behavior labeling model to obtain a third behavior labeling result. The candidate behavior labeling model is trained based on the first image to be identified, the second behavior labeling result and the third behavior labeling result until the candidate behavior labeling model converges to obtain a target behavior labeling model. The target behavior labeling model can be used to implement behavior labeling of target objects in actual scenes. Figure 1 As shown:

[0124] S11: Based on key points in a first picture to be identified, a first line of a target object in the first picture to be identified is obtained as a labeling result.

[0125] Among them, the first picture to be identified refers to a panoramic picture that is not labeled, and the key points refer to some significant feature points on the human body to be identified, which are crucial to determining the position, posture or shape of the human body. In human posture estimation, key points usually refer to the positions of human joints, such as the head, shoulders, elbows, wrists, hips, knees and ankles. The target object is the object that needs to be identified and located in the first picture to be identified. The first behavior annotation result refers to the output result of describing and identifying the movements and postures of the human body by detecting and analyzing key points.

[0126] S12: Based on the first preset model and the first line annotation result, obtain a second line annotation result of the target object in the first to-be-identified picture.

[0127] The first preset model refers to a visual question answering model, which is designed to understand the image content and answer natural language questions related to the image. The second behavior annotation result refers to the behavior annotation result after the behavior classification and verification by the first preset model is verified with the first behavior annotation result.

[0128] S13: Based on the first to-be-recognized image, the second behavior annotation result, and the third behavior annotation result, training a candidate behavior annotation model until the candidate behavior annotation model converges to obtain a target behavior annotation model.

[0129] Among them, the third behavior annotation result is the behavior annotation result obtained by annotating the first picture to be identified based on the candidate behavior annotation model. The candidate behavior annotation model refers to a model that has not been finalized in the behavior annotation task. The candidate behavior annotation model is pre-trained based on a dense crowd data set and can perform preliminary identification and annotation of behaviors in input pictures or video data. The target behavior annotation model refers to the model finally obtained after training and optimization. The target behavior annotation model can accurately identify and annotate behaviors in pictures or video data.

[0130] S14: Annotate the behavior of the target object based on the target behavior annotation model.

[0131] For the trained target behavior labeling model, the target object can be labeled with behavior to achieve efficient detection and recognition of action behaviors.

[0132] In this embodiment, based on the key points in the first picture to be identified, the first behavior annotation result of the target object in the first picture to be identified is obtained; based on the first preset model and the first behavior annotation result, the second behavior annotation result of the target object in the first picture to be identified is obtained; based on the first picture to be identified, the second behavior annotation result and the third behavior annotation result, a candidate behavior annotation model is trained until the candidate behavior annotation model converges to obtain a target behavior annotation model; the behavior of the target object is annotated based on the target behavior annotation model, thereby improving the accuracy of the behavior annotation.

[0133] Figure 2 A flow chart of another target object behavior labeling method provided in an embodiment of the present application: Figure 2 is Figure 1 Based on the embodiment shown, a possible implementation of S11 is further described as follows: Figure 2 As shown:

[0134] S111: Based on the human body detection model, obtain an image of the target object in the first picture to be identified.

[0135] First, you need to prepare a data set containing human targets to train the human detection model. For example, use a dense crowd data set to train the human detection model. Suppose you use the CrowdHuman data set containing high-density and high-occlusion images to train the seventh version of the target detection model (You Only Look Once, YOLO), that is, the YOLO V7 model. The model will learn to recognize and locate the human body from the image, including the head, visible area, and whole body, and mark the human body frame of the person. Use the trained YOLO V7 model to perform human body detection on the classroom data, and the detected student body image is the image of the target object.

[0136] S112: Based on the human posture recognition model, obtain the coordinates of key points in the image of the target object.

[0137] Among them, the key point coordinates include facial key point coordinates and body key point coordinates, the facial key point coordinates include: nose coordinates, first side eye coordinates, second side eye coordinates, first side ear coordinates and second side ear coordinates; the body key point coordinates include: first side shoulder coordinates, second side shoulder coordinates, first side elbow joint coordinates, second side elbow joint coordinates, first side wrist joint coordinates, second side wrist joint coordinates, first side hip coordinates and second side hip coordinates.

[0138] For example, the human pose estimation model is used to detect key points of student human body pictures. Taking the simple deconvolution decoder (Simple Vision Transformer Baselines for Human Pose Estimation, ViTPose) as an example, ViTPose uses a simple visual Transformer as the encoder for feature extraction. The key point prediction is obtained through the decoder, and 17 key points of the human body can be obtained, including: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle.

[0139] S113: Based on the key point coordinates, obtain the head height and head width of the target object.

[0140] Wherein, the head height is obtained based on the coordinates of the facial key points; and the head width is obtained based on the coordinates of the body key points.

[0141] The step of obtaining the head height based on the facial key point coordinates includes:

[0142] The head height is obtained based on the average of the ordinate of the nose coordinate, the ordinate of the first eye coordinate, the ordinate of the second eye coordinate, the ordinate of the first ear coordinate and the ordinate of the second ear coordinate. Wherein, "first" refers to the coordinate located on the same side of the body, and "second" refers to the coordinate located on the other side of the body. In this embodiment, "first" is the left side of the body and "second" is the right side of the body as an example; conversely, "first" can also be the right side of the body and "second" can be the left side of the body, and this application does not impose any mandatory limitation.

[0143] For example, the y coordinate of the nose is nose , the ordinate of the first eye coordinate is y left_eye , the ordinate of the second eye coordinate is y right_eye, The ordinate of the first ear coordinate is y left_ear , the ordinate of the second ear coordinate is yright_ear , then H=(y nose +y left_eye +y right_eye +y left_ear +y right_ear ) / 5 to calculate the head height, where H is the head height.

[0144] The step of obtaining the head width based on the coordinates of key points of the body includes:

[0145] The head width is obtained based on half of the difference between the abscissa of the first shoulder coordinate and the abscissa of the second shoulder coordinate.

[0146] For example, the horizontal coordinate of the first shoulder coordinate is x left_shoulder , the horizontal coordinate of the second shoulder coordinate is x right_shoulder , then we can get W=|x left_shoulder -x right_shoulder | / 2 calculates the head width, where W is the head width.

[0147] S114: Based on the key point coordinates, the head height and the head width, obtain the first line as a labeling result.

[0148] The first line of annotation results includes: the target object is in a first posture, the target object is in a second posture, and the target object is in a third posture. The first posture is "raising hands", the second posture is "lowering the head", and the third posture is "turning around".

[0149] Optionally, if the ordinate of the first-side wrist joint coordinate plus half of the head width is less than the ordinate of the first-side elbow joint coordinate; and the ordinate of the first-side elbow joint coordinate plus half of the head width is less than the ordinate of the first-side shoulder coordinate; and the ordinate of the first-side wrist joint coordinate plus the head width is less than the head height, then it is determined that the first behavior is a labeling result that the target object is in the first posture;

[0150] Alternatively, if the ordinate of the second-side wrist joint coordinate plus half of the head width is smaller than the ordinate of the second-side elbow joint coordinate; and the ordinate of the second-side elbow joint coordinate plus half of the head width is smaller than the ordinate of the second-side shoulder coordinate; and the ordinate of the second-side wrist joint coordinate plus the head width is smaller than the head height, then the first behavior is determined as the annotation result that the target object is in the first posture.

[0151] Optionally, if the ordinate of the first-side wrist joint coordinate plus half of the head width is less than the ordinate of the first-side elbow joint coordinate; and, the ordinate of the first-side elbow joint coordinate plus half of the head width is less than the ordinate of the first-side shoulder coordinate; and, the ordinate of the first-side wrist joint coordinate plus the head width is less than the head height; the ordinate of the second-side wrist joint coordinate plus half of the head width is less than the ordinate of the second-side elbow joint coordinate; and, the ordinate of the second-side elbow joint coordinate plus half of the head width is less than the ordinate of the second-side shoulder coordinate; and, the ordinate of the second-side wrist joint coordinate plus the head width is less than the head height, then it is determined that the annotation result of the first row is that the target object is in the second posture.

[0152] For example, assume that the ordinate of the first-side wrist joint coordinate obtained is y left_wrist , the ordinate of the second-side wrist joint coordinate is y right_wrist , the ordinate of the first-side elbow joint coordinate is y left_elbow , the ordinate of the second-side elbow joint coordinate is y right_elbow , the ordinate of the first-side shoulder coordinate is y left_shoulder , the ordinate of the second-side shoulder coordinate is y right_shoulder . Determine the height relationship between the ordinate of the side wrist joint coordinate, the ordinate of the elbow joint coordinate, and the shoulder coordinate, so as to determine the behavior of the target object. If y left_wrist +W / 2 < y left_elbow and y left_elbow +W / 2 < y left_shoulder and y left_wrist +W < H, it is determined that the annotation result of the first row is that the target object is in the first posture, and at this time it is "raising the hand"; if y right_wrist +W / 2 < y right_elbow and y right_elbow+ W / 2 < y right_shoulder and y right_wrist +W < H, it is determined that the annotation result of the first row is that the target object is in the first posture, and at this time it is "raising the hand";

[0153] If both y left_wrist +W / 2 < y left_elbow and y left_elbow +W / 2 < y left_shoulder and y left_wrist +W < H; and y right_wrist +W / 2 < y right_elbow and y right_elbow +W / 2 < y right_shoulder and y right_wrist +W < H are satisfied, it is determined that the annotation result of the first row is that the target object is in the second posture, and at this time it is "lowering the head". Because in this example, the coordinate system adopted in image processing is as Figure 3The coordinate system shown in the figure, where the x-axis is the horizontal coordinate representing the width, and the y-axis is the vertical coordinate representing the height. The origin of the coordinate is located in the upper left corner of the image, so the higher the position, the smaller the vertical coordinate. Figure 3 A schematic diagram of a coordinate axis provided in an embodiment of the present application.

[0154] Optionally, if at least four of the following conditions are met, determining that the first behavior is labeled as the target object being in a third posture;

[0155] The abscissa of the eye coordinate on the first side is less than the abscissa of the eye coordinate on the second side;

[0156] The abscissa of the first side ear coordinate is smaller than the abscissa of the second side ear coordinate;

[0157] The abscissa of the first side shoulder coordinate is smaller than the abscissa of the second side shoulder coordinate;

[0158] The abscissa of the first side elbow joint coordinate is smaller than the abscissa of the second side elbow joint coordinate;

[0159] The abscissa of the first-side wrist joint coordinate is smaller than the abscissa of the second-side wrist joint coordinate;

[0160] The abscissa of the first side hip coordinate is smaller than the abscissa of the second side hip coordinate.

[0161] For example, traverse 6 pairs of symmetrical parts, including: eyes, ears, shoulders, elbows, wrists and hips, and check whether the x coordinate of the left part is smaller than the x coordinate of the right part. The corresponding relationship of the symmetrical parts is as follows: left eye and right eye, left ear and right ear, left shoulder and right shoulder, left elbow and right elbow, left wrist and right wrist, left hip and right hip. The horizontal coordinate of the eye coordinate on the first side is x left_eye , the horizontal coordinate of the eye coordinate of the second side is x right_eye , the horizontal coordinate of the first ear is x left_ear , the horizontal coordinate of the second ear is x right_ear , the horizontal coordinate of the first shoulder coordinate is x left_shoulder , the horizontal coordinate of the second shoulder coordinate is x right_shoulder , the horizontal coordinate of the first elbow joint is x left_elbow , the horizontal coordinate of the second elbow joint is x right_elbow , the horizontal coordinate of the first wrist joint is x left_wrist , the horizontal coordinate of the second wrist joint is x right_wrist , the horizontal coordinate of the first side hip coordinate is x left_buttocks , the horizontal coordinate of the hip coordinate of the second side is x right_buttocks For each pair of symmetrical parts, if x left <x right, then the flag is incremented by 1. When the calculation of the 6 pairs of symmetrical parts is completed, if the flag is greater than 3, it is determined that the first line is marked as the target object in the first posture, which is a turn. left_eye <x right_eye ;x left_ear <x right_ear ;x left_shoulder <x right_shoulder ;x left_elbow <x right_elbow ;x left_wrist >x right_wrist ;x left_buttocks >x right_buttocks At this time, flag=4>3, and it can be determined that the first line is labeled as the target object is in the third posture, which is "turning around".

[0162] In this embodiment, an image of a target object in a first picture to be identified is acquired through a human body detection model; then, a human body posture recognition model is used according to the target object to acquire the coordinates of key points in the image of the target object; based on the key point coordinates, the head height and the head width of the target object are acquired; finally, based on the key point coordinates, the first line annotation result is acquired, which provides a basis for acquiring the second line annotation result and improves the accuracy of data annotation.

[0163] The following is a description of the teaching scenario as an example. Figure 4 As shown, Figure 4 A flowchart of another method for labeling behavior of a target object provided in an embodiment of the present application is as follows: Figure 4 is Figure 1 or Figure 2 Based on the embodiment shown, a possible implementation of S12 is further described as follows: Figure 4 As shown:

[0164] S121: Input the first image to be identified into a first preset model to obtain a second candidate annotation result.

[0165] The second candidate annotation result is the behavior result annotated after the behavior classification is performed by the first preset model. A possible implementation method of this step is as follows: Figure 5 As shown, Figure 5 A flowchart of another method for labeling behavior of a target object provided in an embodiment of the present application is as follows:

[0166] S1211: Input the first image to be recognized into a visual question answering model.

[0167] Among them, the Visual Question Answering (VQA) model is a cross-modal artificial intelligence model that is specially designed to understand and process questions that contain both image and text information. Specifically, it provides the most appropriate answer based on the given picture and question.

[0168] S1212: Obtain a second initial annotation result obtained in the classification prompt step of the visual question answering model, and determine a second candidate annotation result according to a confidence level and a preset threshold.

[0169] Among them, the second initial labeling result refers to the labeling result after the initial classification using the first preset model, that is, the labeling result of the initial classification using the VQA model; confidence refers to the probability value assigned by the model to the prediction result of each category in a multi-class classification problem, and the probability value reflects the certainty or trust of the model in its prediction; the preset threshold is used to judge whether the model's prediction is reliable enough.

[0170] For example, to classify student actions in classroom pictures, we can divide student behaviors into "raising hand", "turning around", "standing", "sleeping", "playing with mobile phone", "reading and writing", "listening (students sitting or standing normally)", and "blackboard writing". The visual question-answering model divides student actions into 8 categories and can identify student behaviors in more classroom scenarios.

[0171] First, deploy the VQA model, and construct the "prompt word classification" for the VQA model classification task based on the target category, that is, construct prompt_classify. In the VQA model classification task, prompt_classify is a prompt word or instruction used to guide the model to perform a specific classification. Specifically, it is a natural language description or question used to instruct the model to pay attention to and classify specific actions in the image. According to prompt_classify, the target object (i.e., all student bodies) detected in step S111 is behaviorally classified, and the classification result obtained by prompt_classify is the second initial annotation result. In the VQA task, the model needs to understand the image content and the natural language questions related to it, and give the correct answer. The constructed prompt_classify example is: {query: "The picture shows a student in a classroom. Please tell me which category the student's current action belongs to." system: "Action categories include [1. Raising hands, 2. Turning around, 3. Standing, 4. Sleeping, 5. Playing with mobile phones, 6. Reading and writing, 7. Listening to lectures (students sitting or standing normally), 8. Blackboard writing]. Please reply directly with the number corresponding to the category, and do not reply with redundant information."}. Assume that a panoramic picture of a classroom is input. For the detected student body, the input model performs behavior classification and outputs "6", which means that the student is "reading and writing".

[0172] After obtaining the second initial annotation result, obtaining a second candidate annotation result according to the confidence level and the preset threshold value. Specifically, if the confidence level of the second initial annotation result is greater than or equal to the preset threshold value in the classification confirmation step of the visual question answering model, determining the second initial annotation result as the second candidate annotation result;

[0173] If the confidence of the second initial annotation result is less than the preset threshold in the classification confirmation step of the visual question answering model, a manual annotation result is obtained and the manual annotation result is determined to be a second candidate annotation result.

[0174] For example, for the second initial annotation result obtained in the above embodiment, a "prompt word confirmation" for a binary classification task for secondary verification is constructed, that is, prompt_confirm is constructed. prompt_confirm is a mechanism for verifying the results obtained by performing a classification task through prompt_classify in a VQA model. According to prompt_confirm, the second initial annotation result is secondary verified, and the confidence conf after softmax indicating the confirmed token "yes" is output. If conf≥θ, the second initial annotation result is considered to be correctly classified. If conf<θ, the data is further manually verified. Among them, conf is the confidence, θ is the preset threshold, and token refers to an element in the text, which can be a word, character or other language unit. Here, "yes" means an affirmative answer. The constructed prompt_confirm example is: {query: "The picture is a student in a classroom scene. Please ask if the student is {second initial annotation result}." system: "Directly reply 'yes' or 'no', do not reply with redundant information."}. Assuming that for the behavior result with the second initial classification result of "raising hand", the output after secondary classification using prompt_confirm is "yes", then the behavior result with the second initial classification result of "raising hand" is determined as the second candidate labeling result.

[0175] S122: Based on the consistency between the first line annotation result and the second candidate annotation result, obtain the second line annotation result as the annotation result.

[0176] Specifically, if the first behavior annotation result and the second candidate annotation result are consistent, determining that the first behavior is the annotation result or the second candidate annotation result is the second behavior annotation result;

[0177] If the first line annotation result is inconsistent with the second candidate annotation result, a manual annotation result is obtained, and the manual annotation result is determined to be the second line annotation result.

[0178] That is, if the classification results in step S11 are the student bodies of the first and third postures, that is, the first behavior in step S11 is labeled as "raising hands" and "turning around", and is also labeled as "raising hands" and "turning around" in the second candidate labeling results obtained in step S1212, then the behavior is determined as the second behavior labeling result; if the first behavior in step S11 is labeled as "raising hands" and "turning around", and is not labeled as "raising hands" and "turning around" in the second candidate labeling results obtained in step S1212, then the behavior is manually labeled. For example, a panoramic picture of a classroom is input, and for the detected student bodies, the first behavior in step S11 is labeled as "raising hands", and the second candidate labeling result in step S1212 is "standing". At this time, the behavior of the student body is manually verified, and the manual verification result is determined as the second behavior labeling result.

[0179] In this embodiment, by inputting the first picture to be identified into the first preset model, the second initial annotation result obtained in the classification prompt step of the visual question answering model is obtained, and the second candidate annotation result is determined according to the confidence level and the preset threshold value; based on the consistency between the first behavior annotation result and the second candidate annotation result, the second behavior annotation result is obtained; thereby, the accuracy and diversity of data annotation are improved.

[0180] Figure 6 A flowchart of another method for labeling behavior of a target object provided in an embodiment of the present application is as follows: Figure 6 is Figure 1 , Figure 2 or Figure 4 Based on the embodiment shown, a possible implementation of S13 is further described as follows: Figure 6 As shown:

[0181] S131: Use the candidate behavior annotation model to annotate the first image to be identified, and obtain a third behavior as an annotation result.

[0182] For example, the eighth version of the target detection model (You Only Look Once, YOLO), i.e., the YOLO V8 model, is trained using the CrowdHuman dataset, and the trained YOLO V8 model is the candidate image annotation model. The first image to be identified is input into the candidate image annotation model to obtain the annotation result of the candidate image annotation model, and the annotation result is the third line as the annotation result.

[0183] S132: Map the second behavior annotation result back to the first original image to be identified according to the human body detection frame coordinates of S111, and train a candidate behavior annotation model.

[0184] Specifically, the second behavior annotation result is mapped back to the original image of the first image to be identified according to the human body detection frame coordinates of S111 to obtain complete behavior annotation information of each first image to be identified. The first image to be identified with the second behavior annotation result is input into the candidate behavior annotation model, and the model weight, learning rate, etc. are adjusted to enable the candidate image annotation model to better identify the student's behavior.

[0185] S133: Based on the second behavior labeling result and the third behavior labeling result, a target behavior labeling result is obtained, and a target behavior labeling model is obtained using the target behavior labeling result candidate behavior labeling model.

[0186] Specifically, if the second behavior labeling result is consistent with the third behavior labeling result, the second behavior labeling result or the third behavior labeling result is used as the target behavior labeling result, and based on the target behavior labeling result, the candidate behavior labeling model is trained until the candidate behavior labeling model converges to obtain the target behavior labeling model;

[0187] If the second behavior labeling result is inconsistent with the third behavior labeling result, the manual labeling result is used as the target behavior labeling result, and the candidate behavior labeling model is trained based on the target behavior labeling result until the candidate behavior labeling model converges to obtain the target behavior labeling model.

[0188] In this embodiment, the first image to be identified is annotated using the candidate behavior annotation model to obtain the third behavior annotation result; the second behavior annotation result is mapped back to the original image of the first image to be identified according to the human body detection frame coordinates of S111 to train the candidate behavior annotation model; based on the second behavior annotation result and the third behavior annotation result, the target behavior annotation result is obtained, and the target behavior annotation model is obtained using the candidate behavior annotation model of the target behavior annotation result. Herein, steps S132 and S133 are repeated, and the target behavior annotation result of each annotation is added to S133 until all the first images to be identified are annotated, and the candidate behavior annotation model is used to train the target behavior annotation model to improve the prediction accuracy of the model.

[0189] Figure 7 A flowchart of another target object behavior labeling method provided in an embodiment of the present application is shown as follows: Figure 7 As shown, a possible implementation of S114 is as follows:

[0190] S114': Obtain the labeling results of the first line as "raise hand" and "turn around" based on the key point coordinates.

[0191] Whether it is "raising hand" and "turning around" is determined based on the obtained key point coordinates, as described in S114 above, and will not be repeated here.

[0192] like Figure 7 As shown, a possible implementation of S1212 is as follows:

[0193] S12121: Obtaining second candidate annotation results of “raise hand” and “turn around” based on the first preset model.

[0194] The target object obtained in the first preset picture is annotated using the visual question answering model, and the behavior annotation results of "raising hand" and "turning around" are used as the second candidate annotation results. The implementation method of using the visual question answering model for annotation is as described in S1212 above, and will not be repeated here.

[0195] Another possible implementation of S12121 is:

[0196] S12121': Based on the first preset model, the second behavior labeling results are obtained as "standing", "sleeping", "playing with mobile phone", "reading and writing", "listening (students sit normally)" and "blackboard writing".

[0197] The target object obtained in the first preset picture is annotated using the visual question answering model, and the behavior annotation results of "standing", "sleeping", "playing with mobile phone", "reading and writing", "listening (students sit normally)" and "blackboard writing" are used as the second behavior annotation results. The implementation method of using the visual question answering model for annotation is as described in S1212 above, and will not be repeated here.

[0198] Another possible implementation of S122 is:

[0199] S122': based on the consistency between "raise hand" and "turn around" in the first behavior annotation result and "raise hand" and "turn around" in the second candidate annotation result, obtain the second behavior annotation result.

[0200] The behavior whose first behavior is labeled as "raise hand" and "turn around" in S114' is compared with the behavior whose second candidate behavior is labeled as "raise hand" and "turn around" in S12121. If the labeling results are consistent, the first behavior labeling result or the second candidate labeling result is used as the second behavior labeling result; if the labeling results are inconsistent, manual verification is performed and the verified labeling result is used as the second behavior labeling result.

[0201] A possible implementation of S133 is as follows:

[0202] S1331: Obtain a target behavior labeling result based on the consistency between the second behavior labeling result and the third behavior labeling result.

[0203] The implementation method of this step is as described in S133 above and will not be repeated here.

[0204] S1332: Based on the target behavior labeling result, train a candidate behavior labeling model.

[0205] The obtained target behavior labeling results are used as the training set to train the candidate behavior labeling model.

[0206] S1333: Use a small amount of target behavior annotation results to fine-tune the candidate behavior annotation model to obtain a new target behavior annotation result.

[0207] The obtained target behavior labeling result is used as the training set to fine-tune the candidate behavior labeling model, and the labeling result obtained by the candidate behavior labeling model is used as the third behavior labeling result to execute S1331. The target behavior labeling result obtained in S1331 is used again to train the candidate behavior labeling model until the candidate behavior labeling model converges.

[0208] In this embodiment, the target object of the first image to be identified is obtained through a human body detection model; the key points of the target object are obtained based on the human body posture estimation model, and the "hand-raising" and "turning around" in the first behavior annotation result are obtained based on the coordinates of the key points; the second candidate annotation result is obtained based on the visual question-answering model; the "hand-raising" and "turning around" behaviors in the first behavior annotation result and the second candidate annotation result are compared, and if the comparison results are consistent, the first behavior annotation result or the second candidate annotation result is used as the second behavior annotation result; if the comparison results are inconsistent, manual verification is performed, and the verification result is used as the second behavior annotation result; the second behavior annotation result and the third behavior annotation result are compared to obtain the target behavior annotation result, and the obtained target behavior annotation result is used as a training set to fine-tune the candidate behavior annotation model, and the annotation result obtained by the candidate behavior annotation model is used as the third behavior annotation result to be compared again with the second behavior annotation result, and the candidate behavior annotation model is trained with the compared target behavior annotation result until the candidate behavior annotation model converges to obtain the target behavior annotation model. The target behavior annotation model can be used for efficient detection and recognition of student behaviors. The multiple methods used to jointly annotate target behaviors are an end-to-end solution that can directly extract behavioral information from images without the need for additional post-processing steps. This improves reliability and efficiency compared to traditional non-end-to-end methods.

[0209] Figure 8A schematic diagram of the structure of a target object behavior labeling device provided in an embodiment of the present application includes: a first processing module 801, a second processing module 802, a third processing module 803 and a labeling module 804. Among them, the first processing module 801 is used to obtain the first behavior labeling result of the target object in the first picture to be identified based on the key points in the first picture to be identified; the second processing module 802 is used to obtain the second behavior labeling result of the target object in the first picture to be identified based on the first preset model and the first behavior labeling result; the third processing module 803 is used to train a candidate behavior labeling model based on the first picture to be identified, the second behavior labeling result and the third behavior labeling result until the candidate behavior labeling model converges to obtain a target behavior labeling model, wherein the third behavior labeling result is a behavior labeling result obtained by labeling the first picture to be identified based on the candidate behavior labeling model; the labeling module 804 is used to label the behavior of the target object based on the target behavior labeling model.

[0210] Optionally, the first processing module 801 is specifically configured to obtain an image of a target object in the first picture to be identified based on a human body detection model;

[0211] Based on a human posture recognition model, obtaining key point coordinates in the image of the target object;

[0212] Based on the key point coordinates, obtaining the head height and head width of the target object;

[0213] Based on the key point coordinates, the head height and the head width, a first line is obtained as a labeling result.

[0214] Optionally, the key point coordinates include: facial key point coordinates and body key point coordinates;

[0215] Optionally, acquiring the head height and the head width based on the key point coordinates includes: acquiring the head height based on the facial key point coordinates; acquiring the head width based on the body key point coordinates.

[0216] Optionally, the facial key point coordinates include:

[0217] nose coordinates, first-side eye coordinates, second-side eye coordinates, first-side ear coordinates, and second-side ear coordinates;

[0218] Optionally, the first processing module 801 is further configured to obtain the head height based on the facial key point coordinates, including:

[0219] The head height is obtained based on an average value of the ordinate of the nose coordinate, the ordinate of the first eye coordinate, the ordinate of the second eye coordinate, the ordinate of the first ear coordinate, and the ordinate of the second ear coordinate.

[0220] Optionally, the body key point coordinates include: first side shoulder coordinates and second side shoulder coordinates;

[0221] Optionally, the first processing module 801 is further used to obtain the head width based on the coordinates of the key points of the body, including:

[0222] The head width is obtained based on half of the difference between the abscissa of the first shoulder coordinate and the abscissa of the second shoulder coordinate.

[0223] Optionally, the body key point coordinates further include: first side elbow joint coordinates, second side elbow joint coordinates, first side wrist joint coordinates, and second side wrist joint coordinates;

[0224] Optionally, the first processing module 801 is further used to obtain a first behavior annotation result based on the key point coordinates, the head height and the head width, including: if the ordinate of the first-side wrist joint coordinate plus half of the head width is less than the ordinate of the first-side elbow joint coordinate; and the ordinate of the first-side elbow joint coordinate plus half of the head width is less than the ordinate of the first-side shoulder coordinate; and the ordinate of the first-side wrist joint coordinate plus the head width is less than the head height, then determining that the first behavior annotation result is that the target object is in the first posture;

[0225] or,

[0226] If the ordinate of the second wrist joint coordinate plus half of the head width is less than the ordinate of the second elbow joint coordinate; and the ordinate of the second elbow joint coordinate plus half of the head width is less than the ordinate of the second shoulder coordinate; and the ordinate of the second wrist joint coordinate plus the head width is less than the head height, then it is determined that the first line is annotated as the target object being in the first posture;

[0227] Optionally, the method further includes:

[0228] If the ordinate of the wrist joint coordinate on the first side plus half the head width is less than the ordinate of the elbow joint coordinate on the first side; and the ordinate of the elbow joint coordinate on the first side plus half the head width is less than the ordinate of the shoulder coordinate on the first side; and the ordinate of the wrist joint coordinate on the first side plus half the head width is less than the head height; the ordinate of the wrist joint coordinate on the second side plus half the head width is less than the ordinate of the elbow joint coordinate on the second side; and the ordinate of the elbow joint coordinate on the second side plus half the head width is less than the ordinate of the shoulder coordinate on the second side; and the ordinate of the wrist joint coordinate on the second side plus half the head width is less than the head height, then it is determined that the first behavior is annotated as the target object being in the second posture.

[0229] Optionally, the body key point coordinates further include: first side hip coordinates and second side hip coordinates;

[0230] Optionally, the first processing module 801 is further used to determine that the first behavior is marked as the target object being in a third posture if at least four of the following conditions are met;

[0231] The abscissa of the eye coordinate on the first side is less than the abscissa of the eye coordinate on the second side;

[0232] The abscissa of the first side ear coordinate is smaller than the abscissa of the second side ear coordinate;

[0233] The abscissa of the first side shoulder coordinate is smaller than the abscissa of the second side shoulder coordinate;

[0234] The abscissa of the first side elbow joint coordinate is smaller than the abscissa of the second side elbow joint coordinate;

[0235] The abscissa of the first-side wrist joint coordinate is smaller than the abscissa of the second-side wrist joint coordinate;

[0236] The abscissa of the first side hip coordinate is smaller than the abscissa of the second side hip coordinate.

[0237] Optionally, the second processing module 802 is specifically configured to input the first image to be identified into a visual question answering model; and obtain a second initial annotation result obtained in the classification prompting step of the visual question answering model;

[0238] If the confidence of the second initial annotation result is greater than or equal to a preset threshold in the classification confirmation step of the visual question answering model, determining the second initial annotation result as a second candidate annotation result;

[0239] If the confidence of the second initial annotation result is less than the preset threshold in the classification confirmation step of the visual question answering model, a manual annotation result is obtained and the manual annotation result is determined to be a second candidate annotation result.

[0240] Optionally, the second processing module 802 is further configured to obtain the second line annotation result based on the consistency between the first line annotation result and the second candidate annotation result, including:

[0241] If the first behavior annotation result and the second candidate annotation result are consistent, determining that the first behavior is the annotation result or the second candidate annotation result is the second behavior annotation result;

[0242] If the first line annotation result is inconsistent with the second candidate annotation result, a manual annotation result is obtained, and the manual annotation result is determined to be the second line annotation result.

[0243] Optionally, the third processing module 803 is specifically used to train a candidate behavior annotation model based on the first to-be-recognized image, the second behavior annotation result, and the third behavior annotation result until the candidate behavior annotation model converges to obtain a target behavior annotation model, including:

[0244] If the second behavior labeling result is consistent with the third behavior labeling result, the second behavior labeling result or the third behavior labeling result is used as the target behavior labeling result, and based on the target behavior labeling result, the candidate behavior labeling model is trained until the candidate behavior labeling model converges to obtain the target behavior labeling model;

[0245] If the second behavior labeling result is inconsistent with the third behavior labeling result, the manual labeling result is used as the target behavior labeling result, and the candidate behavior labeling model is trained based on the target behavior labeling result until the candidate behavior labeling model converges to obtain the target behavior labeling model.

[0246] The target object behavior labeling method and the target object behavior labeling device in the embodiments of the present application can be used to implement the technical solutions of the above-mentioned method embodiments. Their implementation principles and technical effects are similar and will not be repeated here.

[0247] The present application also provides an electronic device, the electronic device comprising a processor and a memory. The memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the following is achieved: Figures 1 to 7 The steps of the target object behavior labeling method are shown.

[0248] The embodiment of the present application also provides a computer-readable storage medium, wherein a program or instruction is stored on the computer-readable storage medium, and when the program or instruction is executed by a processor, the following is implemented: Figures 1 to 7 The steps of the target object behavior labeling method are shown.

[0249] The present application also provides a computer program product, which, when executed by a processor of a vehicle or a cloud server, implements the following Figures 1 to 7 The steps of the target object behavior labeling method are shown.

[0250] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable a terminal or a network-side device to execute the methods described in each embodiment of the present application.

[0251] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A method for labeling a target object's behavior, characterized in that: include: Based on the key points in the first picture to be identified, obtaining a labeling result for a first line of the target object in the first picture to be identified; Based on the first preset model and the first line annotation result, obtaining a second line annotation result of the target object in the first to-be-identified picture; Based on the first image to be identified, the second behavior annotation result, and the third behavior annotation result, training a candidate behavior annotation model until the candidate behavior annotation model converges to obtain a target behavior annotation model, wherein the third behavior annotation result is a behavior annotation result obtained by annotating the first image to be identified based on the candidate behavior annotation model; The target object’s behavior is labeled based on the target behavior labeling model.

2. The method according to claim 1, characterized in that: The training of the candidate behavior annotation model based on the first to-be-recognized image, the second behavior annotation result, and the third behavior annotation result until the candidate behavior annotation model converges to obtain a target behavior annotation model includes: If the second behavior labeling result is consistent with the third behavior labeling result, the second behavior labeling result or the third behavior labeling result is used as the target behavior labeling result, and based on the target behavior labeling result, the candidate behavior labeling model is trained until the candidate behavior labeling model converges to obtain the target behavior labeling model; If the second behavior labeling result is inconsistent with the third behavior labeling result, the manual labeling result is used as the target behavior labeling result, and the candidate behavior labeling model is trained based on the target behavior labeling result until the candidate behavior labeling model converges to obtain the target behavior labeling model.

3. The method according to claim 2, characterized in that The step of obtaining the second behavior annotation result of the target object in the first to-be-identified picture based on the first preset model and the first behavior annotation result includes: Inputting the first image to be identified into a first preset model to obtain a second candidate annotation result; Based on the consistency between the first line annotation result and the second candidate annotation result, the second line annotation result is obtained.

4. The method according to claim 3, characterized in that The obtaining the second line annotation result based on the consistency between the first line annotation result and the second candidate annotation result includes: If the first behavior annotation result and the second candidate annotation result are consistent, determining that the first behavior is the annotation result or the second candidate annotation result is the second behavior annotation result; If the first line annotation result is inconsistent with the second candidate annotation result, a manual annotation result is obtained, and the manual annotation result is determined to be the second line annotation result.

5. The method according to claim 4, characterized in that The step of inputting the first image to be identified into a first preset model to obtain a second candidate labeling result includes: Based on inputting the first picture to be recognized into a visual question answering model; Obtaining a second initial annotation result obtained in the classification prompt step of the visual question answering model; If the confidence of the second initial annotation result is greater than or equal to a preset threshold in the classification confirmation step of the visual question answering model, determining the second initial annotation result as a second candidate annotation result; If the confidence of the second initial annotation result is less than the preset threshold in the classification confirmation step of the visual question answering model, a manual annotation result is obtained and the manual annotation result is determined to be a second candidate annotation result.

6. The method according to claim 1, characterized in that The step of obtaining the first line of the target object in the first picture to be identified as a labeling result based on the key points in the first picture to be identified includes: Based on the human body detection model, obtaining an image of the target object in the first picture to be identified; Based on a human posture recognition model, obtaining key point coordinates in the image of the target object; Based on the key point coordinates, obtaining the head height and head width of the target object; Based on the key point coordinates, the head height and the head width, a first line is obtained as a labeling result.

7. The method according to claim 6, characterized in that The key point coordinates include: facial key point coordinates and body key point coordinates; The step of obtaining the head height and the head width based on the key point coordinates includes: Based on the coordinates of facial key points, obtaining the head height; Based on the coordinates of the key points of the body, the head width is obtained.

8. The method according to claim 7, characterized in that The facial key point coordinates include: nose coordinates, first-side eye coordinates, second-side eye coordinates, first-side ear coordinates, and second-side ear coordinates; The step of obtaining the head height based on the facial key point coordinates includes: The head height is obtained based on an average value of the ordinate of the nose coordinate, the ordinate of the first eye coordinate, the ordinate of the second eye coordinate, the ordinate of the first ear coordinate, and the ordinate of the second ear coordinate.

9. The method according to claim 7, characterized in that: The body key point coordinates include: first side shoulder coordinates and second side shoulder coordinates; The step of obtaining the head width based on the coordinates of key points of the body includes: The head width is obtained based on half of the difference between the abscissa of the first shoulder coordinate and the abscissa of the second shoulder coordinate.

10. The method according to claim 9, characterized in that The body key point coordinates also include: first side elbow joint coordinates, second side elbow joint coordinates, first side wrist joint coordinates, and second side wrist joint coordinates; The obtaining the first line as a labeling result based on the key point coordinates, the head height and the head width includes: If the ordinate of the first wrist joint coordinate plus half of the head width is less than the ordinate of the first elbow joint coordinate; and the ordinate of the first elbow joint coordinate plus half of the head width is less than the ordinate of the first shoulder coordinate; and the ordinate of the first wrist joint coordinate plus the head width is less than the head height, then it is determined that the first line is annotated as the target object being in the first posture; or, If the ordinate of the second-side wrist joint coordinate plus half of the head width is smaller than the ordinate of the second-side elbow joint coordinate; and the ordinate of the second-side elbow joint coordinate plus half of the head width is smaller than the ordinate of the second-side shoulder coordinate; and the ordinate of the second-side wrist joint coordinate plus the head width is smaller than the head height, then the first behavior is determined as the annotation result that the target object is in the first posture.

11. The method according to claim 10, characterized in that The method further comprises: If the ordinate of the wrist joint coordinate on the first side plus half the head width is less than the ordinate of the elbow joint coordinate on the first side; and the ordinate of the elbow joint coordinate on the first side plus half the head width is less than the ordinate of the shoulder coordinate on the first side; and the ordinate of the wrist joint coordinate on the first side plus half the head width is less than the head height; the ordinate of the wrist joint coordinate on the second side plus half the head width is less than the ordinate of the elbow joint coordinate on the second side; and the ordinate of the elbow joint coordinate on the second side plus half the head width is less than the ordinate of the shoulder coordinate on the second side; and the ordinate of the wrist joint coordinate on the second side plus half the head width is less than the head height, then it is determined that the first behavior is annotated as the target object being in the second posture.

12. The method according to claim 11, characterized in that The body key point coordinates also include: first side hip coordinates and second side hip coordinates; The method further comprises: If at least four of the following conditions are met, determining that the first behavior is labeled as the target object being in a third posture; The abscissa of the eye coordinate on the first side is less than the abscissa of the eye coordinate on the second side; The abscissa of the first side ear coordinate is smaller than the abscissa of the second side ear coordinate; The abscissa of the first side shoulder coordinate is smaller than the abscissa of the second side shoulder coordinate; The abscissa of the first side elbow joint coordinate is smaller than the abscissa of the second side elbow joint coordinate; The abscissa of the first-side wrist joint coordinate is smaller than the abscissa of the second-side wrist joint coordinate; The abscissa of the first side hip coordinate is smaller than the abscissa of the second side hip coordinate.

13. A target object behavior labeling device, characterized in that: include: A first processing module, configured to obtain a labeling result of a first line of a target object in a first picture to be identified based on key points in the first picture to be identified; A second processing module, configured to obtain a second line annotation result of the target object in the first to-be-identified picture based on the first preset model and the first line annotation result; a third processing module, configured to train a candidate behavior annotation model based on the first image to be identified, the second behavior annotation result, and the third behavior annotation result, until the candidate behavior annotation model converges to obtain a target behavior annotation model, wherein the third behavior annotation result is a behavior annotation result obtained by annotating the first image to be identified based on the candidate behavior annotation model; The labeling module is used to label the behavior of the target object based on the target behavior labeling model.

14. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the target object behavior labeling method as described in any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the target object behavior labeling method according to any one of claims 1 to 12 are implemented.

Citation Information

Patent Citations

  • Model training method, customer service system, data labeling system and readable storage medium

    CN109582793A

  • Multi-human-body posture detection and state discrimination method based on instance segmentation

    CN111563452A

  • Image annotation verification method and device based on artificial intelligence, equipment and medium

    CN113705691A

  • Behavior recognition method and device and storage medium

    CN114202797A

  • Method for cooperatively analyzing and identifying abnormal behaviors of examinees based on multiple cameras

    CN115035433A