Image annotation method, device, electronic device and storage medium

Abstract

The present application proposes an image annotation method, device, electronic device and storage medium, the image annotation method comprising: performing posture estimation processing on the image to be annotated to obtain first posture data of the target in the image to be annotated; determining second posture data from the first posture data; generating a first image containing the target based on the second posture data and the image to be annotated; generating a second image not containing the target based on the image to be annotated; performing depth estimation on the image to be annotated to obtain depth information of the target; generating an annotated image based on the depth information, the first image and the second image. The present application relates to the field of artificial intelligence technology and can improve the accuracy of image annotation.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image annotation method, device, electronic device and storage medium. Background Art

[0002] At present, with the development of artificial intelligence technology, more and more organizations or individuals tend to use artificial intelligence models to assist production in order to improve productivity. In order to use artificial intelligence models to process surveillance videos collected by cameras, some organizations or individuals usually need to first train the artificial intelligence models based on image data with annotation information to improve the performance of the artificial intelligence models. For example, in public areas, in order to use artificial intelligence models to monitor the actions of targets in videos for analysis to identify conflicting actions of targets in public areas, images annotated with human posture information can be used to train artificial intelligence models for action recognition, thereby improving the accuracy of action recognition. In order to improve the efficiency of image annotation, targets in images are usually annotated manually, which has the problem of low efficiency and low accuracy of image annotation. Summary of the invention

[0003] In view of the above, it is necessary to propose an image annotation method, device, electronic device and storage medium to solve the technical problem of low accuracy when annotating images.

[0004] The present application provides an image annotation method, which is applied to an electronic device. The method includes: performing posture estimation processing on an image to be annotated to obtain first posture data of a target in the image to be annotated; determining second posture data from the first posture data; generating a first image containing the target according to the second posture data and the image to be annotated; generating a second image not containing the target according to the image to be annotated; wherein the second image includes background information in the image to be annotated; performing depth estimation on the image to be annotated to obtain depth information of the target; and generating an annotated image according to the depth information, the first image, and the second image.

[0005] The embodiment of the present application also provides an image annotation device, which includes: a posture estimation module, which is used to perform posture estimation processing on the image to be annotated to obtain first posture data of the target in the image to be annotated; the posture estimation module is also used to determine second posture data from the first posture data; a generation module, which is used to generate a first image containing the target according to the second posture data and the image to be annotated; the generation module is also used to generate a second image that does not contain the target according to the image to be annotated; wherein the second image includes background information in the image to be annotated; a depth estimation module, which is used to perform depth estimation on the image to be annotated to obtain depth information of the target; and an annotation module, which is used to generate an annotated image according to the depth information, the first image and the second image.

[0006] An embodiment of the present application further provides an electronic device, comprising: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the image annotation method.

[0007] An embodiment of the present application further provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in an electronic device to implement the image annotation method.

[0008] It can be seen from the above technical solutions that the embodiment of the present application first performs posture estimation processing on the image to be annotated, determines the first posture data of the target in the image to be annotated, and determines the second posture data from the first posture data according to the score of the first posture data, so that the first posture data with a low score can be screened out, thereby improving the accuracy of subsequent image annotation. Then, a first image containing the target is generated according to the second posture data and the image to be annotated, so that the quality of the posture information corresponding to the target can be improved by reconstructing the posture of the target in the image to be annotated. A second image that does not contain the target is also generated according to the image to be annotated, so that the background information in the image to be annotated is represented by the second image, and the target and background in the image to be annotated can be separated before the image is annotated, thereby reducing the error caused by the background information in the image annotation process. And the depth of the image to be annotated is estimated to determine the depth information of the target, and finally annotated images are generated according to the depth information, the first image and the second image, and the depth information is used to represent the positional relationship between different targets, which can improve the accuracy of the target posture annotation. And the annotation process does not require manual intervention, and a large number of images can be quickly annotated, thereby improving the efficiency of image annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is an application scenario diagram of an image annotation method provided in an embodiment of the present application.

[0010] Figure 2 This is a flowchart of an image annotation method provided by an embodiment of the present application.

[0011] Figure 3 It is a schematic diagram of an image to be annotated provided in an embodiment of the present application.

[0012] Figure 4 This is a flowchart of a method for determining second posture data provided by an embodiment of the present application.

[0013] Figure 5 This is a flowchart of a method for determining a first image provided by an embodiment of the present application.

[0014] Figure 6 is a schematic diagram of a first image provided by an embodiment of the present application.

[0015] Figure 7 is a schematic diagram of a second image provided in an embodiment of the present application.

[0016] Figure 8 This is a flowchart of a method for determining depth information provided by an embodiment of the present application.

[0017] Fig. 9 This is a flowchart of a method for generating annotated images provided in one embodiment of the present application.

[0018] Fig.10 It is a functional module diagram of an image annotation device provided in one embodiment of the present application.

[0019] Fig.11 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth to facilitate a full understanding of the present application, and the embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments.

[0021] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0023] The embodiment of the present application provides an image annotation method, which can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA), a digital processor (Digital Signal Processor, DSP), an embedded device, etc.

[0024] An electronic device can be any electronic product that can interact with customers, such as personal computers, tablet computers, smart phones, personal digital assistants (PDAs), game consoles, Internet Protocol Television (IPTV), smart wearable devices, etc.

[0025] The electronic device may also include a network device and / or a client device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0026] The networks where electronic devices are located include but are not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0027] like Figure 1As shown, an image annotation method provided by the present application can be applied to an electronic device 100, and the electronic device 100 is communicatively connected to a camera device 200, and the electronic device 100 is used to receive a surveillance video sent by the camera device 200. Among them, the camera device 200 is used to shoot targets (for example, pedestrians, animals, plants, vehicles, etc.) within the imaging range to obtain an image to be annotated including the target. A posture estimation model for performing posture estimation on the image to be annotated is run in the electronic device 100, and is used to perform posture estimation on the target in the image to be annotated. The electronic device 100 also scores and screens the first posture data to determine the second posture data, and generates a first image containing the target based on the second posture data and the image to be annotated, and generates a second image not containing the target based on the image to be annotated, so that the target information and background information in the image to be annotated can be separated. And the depth of the image to be annotated is estimated to obtain the depth information of the target, and the annotated image is generated based on the depth information, the first image and the second image. In this way, in the process of annotating the image to be annotated, the first posture data can be scored and screened to determine the second posture data with higher data quality, and the first image can be generated based on the second posture data and the image information of the image to be annotated to reconstruct the target in the first image, thereby improving the image quality of the area containing the target. The first image and the second image are used to separate the target and the background, and finally the depth of the image to be annotated is estimated to obtain the depth information of the target, and then the annotated image is generated based on the depth information, the first image and the second image. The depth information can be used to characterize the positional relationship between different targets, and the accuracy of annotating the target's posture can be improved. In addition, the annotation process does not require manual intervention, and a large number of images can be quickly annotated, thereby improving the efficiency of image annotation.

[0028] like Figure 2 , is a flow chart of an image annotation method provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. An image annotation method provided in an embodiment of the present application includes the following steps.

[0029] S20, performing posture estimation processing on the image to be labeled to obtain first posture data of the target in the image to be labeled.

[0030] In one embodiment of the present application, when annotating the posture of a target in a surveillance video, in order to improve the accuracy of image annotation, a pre-trained target detection model can be used to perform target detection on the video frame to be annotated in the surveillance video, determine the detection frame corresponding to the target in the video frame to be annotated, and then predict the first posture data corresponding to the target in the detection frame according to the posture prediction model. Among them, the image to be annotated can be any video frame in the surveillance video. Specifically, the detection frame is used to characterize the position of the target in the video frame to be annotated; the first posture data includes the coordinates of the first key point and the corresponding confidence. Among them, the first key point is used to characterize the pixel points of the key parts (for example, neck, elbow, wrist, hip, etc.) of the target in the image to be annotated, the coordinates of the first key point can be the coordinates of the first key point in the image coordinate system of the image to be annotated, and the confidence corresponding to the first key point can be the possibility that the pixel point belongs to the key part of the target.

[0031] In one embodiment of the present application, when annotating a target in an image, first predicting the posture data of the target in the image can improve the accuracy of the annotation. By predicting the posture data of the target, the precise position and shape of the target in the image can be obtained, and the boundary of the target to be annotated can be determined more accurately, reducing the annotation error. And the posture data of the target predicted first can provide the spatial structure and motion information of the target in the subsequent posture annotation process, and can more accurately characterize the state of the target in the image to be annotated, thereby improving the accuracy of the annotation. In addition, performing target detection before image annotation can also improve the annotation efficiency. For example, for targets with regular shapes or motion patterns, preliminary annotation results can be generated through target detection and posture prediction, and then the predicted multiple posture data can be screened, thereby improving the efficiency of subsequent image annotation based on posture data. Since the predicted posture data can be used to characterize the preliminary position and posture information of the target, it can reduce the time consumed in determining the target in the image to be annotated during the image annotation process. It can also enhance the flexibility and adaptability of image annotation, and avoid the influence of factors such as occlusion, deformation or lighting changes that the target may be affected by in complex scenes, thereby improving the robustness of the image annotation process in a variety of environments.

[0032] Exemplarily, the target detection model can be a target detection and recognition algorithm based on region proposal, such as convolutional neural network, recurrent convolutional neural network, fast convolutional neural network, etc. The target detection and recognition algorithm based on region proposal first selects the candidate region in the image to be labeled, then uses the convolutional neural network to extract the image features in the image to be labeled, and finally classifies and regresses the image features to determine the detection frame in the image to be labeled, and characterizes the position of the target in the image to be labeled based on the detection frame. The target detection model can also be a target detection and recognition algorithm based on regression, such as the You Only Look Once model and the multi-detection frame recognizer algorithm model. The regression-based target detection and recognition algorithm directly predicts the position and category of the target in the image to be labeled based on the regression algorithm.

[0033] Exemplarily, the posture recognition model can be used to predict the positions of key parts of the target in the image to predict different postures. For example, the posture recognition model can be an open posture model (OpenPose model), which is used to detect key points of the human body such as hands, faces, feet, etc. of multiple people in real-time scenes. The posture recognition model can also be an action network (MoveNet), which is used to detect multiple key points of the human body in the image. The posture recognition model can also be a posture network (PoseNet), which is used to detect the posture of the target in the image to be annotated in real time, and can effectively perform single posture and multi-posture detection of the human body. In addition, the posture network also provides multiple key points for various parts of the human body from the eyes to the ankles, and generates confidence scores to determine the accuracy of the model in identifying specific key points from the image to be annotated.

[0034] In one embodiment of the present application, any one or more video frames in the surveillance video can be used as images to be annotated, so that the overall situation of all targets in the surveillance video can be represented based on one or more video frames, such as the appearance of people, the occurrence of actions, etc. In addition, selecting one or more video frames from the surveillance video as images to be annotated can reduce the complexity of the subsequent image annotation process and improve the efficiency of image annotation. Figure 3The figure shows a schematic diagram of an image 300 to be annotated, wherein the image 300 to be annotated includes a target 310, a target 320, a target 330, and a target 340. The first posture data corresponding to the target 310 includes at least a first key point 311, a first key point 312, and a first key point 313, wherein the confidence of the first key point 311 may be 0.9, the confidence of the first key point 312 may be 0.95, and the confidence of the first key point 313 may be 0.99; the second posture data corresponding to the target 320 includes at least a first key point 321, a first key point 322, and a first key point 323, wherein the confidence of the first key point 321 may be 0.9, the confidence of the first key point 322 may be 0.95, and the confidence of the first key point 323 may be 0.99; The second posture data corresponding to the target 330 includes at least the first key point 331, the first key point 332 and the first key point 333, wherein the confidence of the first key point 331 may be 0.9, the confidence of the first key point 332 may be 0.95, and the confidence of the first key point 333 may be 0.99; the second posture data corresponding to the target 340 includes at least the first key point 341, the first key point 342 and the first key point 343, wherein the confidence of the first key point 341 may be 0.5, the confidence of the first key point 342 may be 0.65, and the confidence of the first key point 343 may be 0.59.

[0035] In this way, the area containing the target (for example, a person) in the image to be annotated and the first posture data corresponding to the target can be obtained, thereby providing data support for the subsequent image annotation process and improving the accuracy and efficiency of image annotation.

[0036] S21, determining second posture data from the first posture data.

[0037] In one embodiment of the present application, in order to improve the accuracy of labeling the target in the image to be labeled, the first posture data can be screened to determine the second posture data with higher accuracy from the first posture data. Specifically, the score of the first posture data can be determined according to the key points and their corresponding confidence levels, and the second posture data can be determined from the first posture data according to the size of the score. Among them, the score of the first posture data can be used to characterize the rationality of the posture of the target predicted by the posture recognition model. Exemplarily, in the case of a lower score of the first posture data, it indicates that the posture represented by the first posture data is more unreasonable, and the accuracy of the posture of the target predicted by the posture recognition model is lower; in the case of a higher score of the first posture data, it indicates that the rationality of the posture represented by the first posture data is higher, and the accuracy of the posture of the target predicted by the posture recognition model is higher.

[0038] Wherein, the score of the first posture data can be a floating point number between 0 and 1, where 1 represents that the first posture data is completely accurate, and 0 represents that the first posture data is completely inaccurate. Exemplarily, the specific method of determining the score of the first posture data includes: matching the predicted first key point with a preset posture template, where the posture template can be a set of all key point positions under the standard posture; for each first key point, calculating the error between the actual position of the first key point and the corresponding key point position in the template, where the error can be the Euclidean distance or the angle deviation; calculating the product of the error corresponding to each first key point and its confidence, and determining the inverse of the product as the score of the first key; determining the mean of the scores corresponding to all first key points, and obtaining the score of the first posture data. The score can represent the degree of difference between the first posture data and the preset posture template. In the case of a higher score, it indicates that the first posture data is more similar to the posture template, the first posture data is more accurate, and the first posture data can be determined as the second posture data; in the case of a lower score, it indicates that the difference between the first posture data and the posture template is greater, and the accuracy of the first posture data is lower.

[0039] In one embodiment of the present application, in order to improve the accuracy of labeling the target in the image to be labeled, the first posture data may be screened to determine the second posture data from the first posture data. Figure 4 The corresponding detailed description.

[0040] S22: Generate a first image including the target according to the second posture data and the image to be annotated.

[0041] In one embodiment of the present application, the posture of some targets in the image to be annotated may be difficult to accurately identify the posture information due to environmental factors. For example, when the target in the image to be annotated is facing the light source, light saturation may occur, resulting in too high image brightness, making it difficult to identify the posture information; when it is facing away from the light source, insufficient light may occur, and the image brightness is too low, which also affects the extraction of posture information. Local saturation or insufficient light will lead to uneven image brightness, and the number and spatial distribution of key points in the posture information will change over time, increasing the difficulty of posture estimation. In the process of collecting the image to be annotated, factors such as external vibration, instrument instability, and shaking of the bracket will cause position measurement errors, thereby affecting the accuracy of posture estimation. Therefore, prompt data can be constructed based on the first posture data of the target in the image to be annotated and the image information of the image to be annotated, and a first image containing the target can be generated based on the prompt data, thereby improving the defects of the posture information in the image to be annotated.

[0042] In this way, a first image with good quality can be generated based on the prompt data containing the target posture and composition information, thereby avoiding the problem of low data annotation accuracy caused by low imaging quality of the image to be annotated. For a specific method of generating a first image containing a target based on the second posture data and the image to be annotated, please refer to Figure 5 The corresponding detailed description.

[0043] In this way, by optimizing the posture and composition of the target, the target is made more prominent in the first image, and the first image can have a higher resolution and clarity, and the first image can be further edited and adjusted according to actual needs. Figure 6 FIG. 6 is a schematic diagram of a first image 600 determined according to the image to be labeled 300 , wherein the first image 600 includes a target 610 , a target 620 , and a target 630 . The second posture data corresponding to the target 610 includes at least the first key point 611, the first key point 612 and the first key point 613, wherein the confidence of the first key point 611 may be 0.9, the confidence of the first key point 612 may be 0.95, and the confidence of the first key point 613 may be 0.99; the second posture data corresponding to the target 620 includes at least the first key point 621, the first key point 622 and the first key point 623, wherein the confidence of the first key point 621 may be 0.9, the confidence of the first key point 622 may be 0.95, and the confidence of the first key point 623 may be 0.99; the second posture data corresponding to the target 630 includes at least the first key point 631, the first key point 632 and the first key point 633, wherein the confidence of the first key point 631 may be 0.9, the confidence of the first key point 632 may be 0.95, and the confidence of the first key point 633 may be 0.99.

[0044] S23, generating a second image that does not include the target according to the image to be annotated; wherein the second image includes background information in the image to be annotated.

[0045] In one embodiment of the present application, in the process of annotating a target in an image to be annotated, in order to improve the quality of image information belonging to the target in the image to be annotated, the background area of ​​the image to be annotated can be reconstructed to generate a second image that does not contain the target, wherein the second image is used to characterize the background information in the image to be annotated. In this way, the annotated image can be synthesized according to the background image and the first image containing the target in the subsequent process. By separating the background and the target, the accuracy of the annotated target area can be improved, thereby avoiding the interference of pixels in the background area, and in the case of complex target boundaries or cluttered backgrounds, it is helpful to improve the accuracy of image annotation. After separating the background and the target, the target can be annotated independently. Since the background image may be relatively simple and stable, while the target image may change due to factors such as posture and position, by reconstructing the background image, the workload of repeatedly annotating the background can be reduced, thereby improving the annotation efficiency. In addition, the target may be blocked or partially blocked by other objects. By reconstructing the background image and combining the target image for synthetic annotation, the situation where the target is blocked can be simulated, thereby performing more accurate annotation. In large-scale image annotation tasks, by reconstructing the background image and using the same background for annotation, the consistency of the annotation results can be ensured, the annotation quality can be improved, and diverse annotation data can be generated, which will help improve the generalization ability of the model when the annotated images are used to train the artificial intelligence model in the future, ensuring that the trained artificial intelligence model can adapt to different application scenarios.

[0046] For example, Figure 7 FIG. 7 is a schematic diagram of a second image 700 determined according to the image to be annotated 300 , wherein the second image 700 includes background information in the image to be annotated 300 , and the second image 700 does not include any object in the image to be annotated 300 .

[0047] S24, performing depth estimation on the image to be labeled to obtain depth information of the target.

[0048] In one embodiment of the present application, in the process of annotating targets in an image to be annotated, in order to refine the position differences between different targets in the image to be annotated, the image to be annotated may be firstly depth estimated to obtain the depth information of each target. By predicting the depth information of the target, the three-dimensional position of the target in the image to be annotated can be determined, thereby improving the accuracy of the annotated target and reducing the annotation errors caused by changes in viewing angle or target overlap. In addition, the depth information can characterize the semantics of the image information, ensuring that the annotation information obtained when annotating complex scenes is robust. In the case of insufficient lighting or partial obstruction of the target, the depth information can help the algorithm better identify and annotate the target, and the use of depth information for annotation can reduce the reliance on manual annotation, improve the degree of automation of annotation, help reduce annotation costs, and improve annotation efficiency.

[0049] In one embodiment of the present application, when multiple objects in an image overlap, the front-to-back relationship between the objects can be determined by predicting the depth information of the objects, so that each object can be labeled more accurately. Figure 8 The corresponding detailed description.

[0050] S25: Generate a labeled image according to the depth information, the first image, and the second image.

[0051] In one embodiment of the present application, spatial information can be provided to the image annotation process based on the depth information, and the first image and the second image can be synthesized based on the depth information to obtain an annotated image. Among them, the depth information can indicate the depth values ​​of different areas in the image, so as to determine the front and back relationship between multiple targets. By judging the position of each target based on the depth value, it can be ensured that the position of the target in the annotated image conforms to the real logic and avoids occlusion errors. The target in the first image is merged into the second image that does not contain the target based on the depth information, and the front and back relationship of different targets can be adjusted based on the depth information to ensure the spatial consistency between the target and the background, and generate an annotated image with posture annotation based on the second posture data.

[0052] In one embodiment of the present application, the annotated image can be used for subsequent computer vision tasks. For example, the annotated image can be used to train an artificial intelligence model with gesture recognition function. Fig. 9 The corresponding detailed description.

[0053] It can be seen from the above technical solutions that the embodiment of the present application first performs posture estimation processing on the image to be annotated, determines the first posture data of the target in the image to be annotated, and determines the second posture data from the first posture data according to the score of the first posture data, so that the first posture data with a low score can be screened out, thereby improving the accuracy of subsequent image annotation. Then, a first image containing the target is generated according to the second posture data and the image to be annotated, so that the quality of the posture information corresponding to the target can be improved by reconstructing the posture of the target in the image to be annotated. A second image that does not contain the target is also generated according to the image to be annotated, so that the background information in the image to be annotated is represented by the second image, and the target and background in the image to be annotated can be separated before the image is annotated, thereby reducing the error caused by the background information in the image annotation process. And the depth of the image to be annotated is estimated to determine the depth information of the target, and finally annotated images are generated according to the depth information, the first image and the second image, and the depth information is used to represent the positional relationship between different targets, which can improve the accuracy of the target posture annotation. And the annotation process does not require manual intervention, and a large number of images can be quickly annotated, thereby improving the efficiency of image annotation.

[0054] like Figure 4 , is a flow chart of a method for determining a preset duration provided by an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining a preset duration provided by an embodiment of the present application includes the following steps.

[0055] S30: Determine a score of the first posture data according to the coordinates of the first key point in the first posture data and the confidence of the first key point.

[0056] In one embodiment of the present application, the score of the first posture data can be determined based on the key points and their corresponding confidence levels, and the second posture data can be determined from the first posture data based on the size of the score. The score of the first posture data can be used to characterize the rationality of the posture of the target predicted by the posture recognition model. Exemplarily, when the score of the first posture data is lower, it indicates that the posture represented by the first posture data is more unreasonable, and the accuracy of the posture of the target predicted by the posture recognition model is lower; when the score of the first posture data is higher, it indicates that the rationality of the posture represented by the first posture data is higher, and the accuracy of the posture of the target predicted by the posture recognition model is higher.

[0057] Among them, the score of the first posture data can be a floating point number between 0 and 1, wherein 1 represents that the first posture data is completely accurate, and 0 represents that the first posture data is completely inaccurate. Exemplarily, the specific method of determining the score of the first posture data includes: matching the predicted first key point with a preset posture template, wherein the posture template can be a set of all key point positions under a standard posture; for each first key point, calculating the error between the actual position of the first key point and the corresponding key point position in the template, wherein the error can be the Euclidean distance or the angle deviation; calculating the product of the error corresponding to each first key point and its confidence, and determining the inverse of the product as the score of the first key; determining the mean of the scores corresponding to all first key points to obtain the score of the first posture data.

[0058] S31, when the score reaches a preset threshold, determining the corresponding first posture data as second posture data.

[0059] In one embodiment of the present application, the score of the first posture data can characterize the degree of difference between the first posture data and a preset posture template. When the score is higher, the first posture data is more similar to the posture template, and the first posture data is more accurate. When the score is higher than a preset threshold, the first posture data can be determined to be the second posture data; when the score is lower, the difference between the first posture data and the posture template is greater, and the accuracy of the first posture data is lower.

[0060] Exemplarily, when the preset threshold is 0.6, if the score of the first posture data is 0.7, it can be determined that the first posture data is the second posture data; if the score of the first posture data is 0.59, it can be determined that the first posture data is not the second posture data.

[0061] like Figure 5 , is a flow chart of a method for determining fusion features provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining fusion features provided in an embodiment of the present application includes the following steps.

[0062] S40, determining prompt data according to the coordinates and confidence of the first key point in the second posture data, the image to be annotated, and a pre-stored prompt template; wherein the prompt data is used to characterize the position information and posture information of the target in the image to be annotated.

[0063] In one embodiment of the present application, the posture of some targets in the image to be annotated may be difficult to accurately identify the posture information due to environmental factors. For example, when the target in the image to be annotated is facing the light source, light saturation may occur, resulting in too high image brightness, making it difficult to identify the posture information; when it is facing away from the light source, insufficient light may occur, and the image brightness is too low, which also affects the extraction of posture information. Local saturation or insufficient light will lead to uneven image brightness, and the number and spatial distribution of key points in the posture information will change over time, increasing the difficulty of posture estimation. In the process of collecting the image to be annotated, factors such as external vibration, instrument instability, and shaking of the bracket will cause position measurement errors, thereby affecting the accuracy of posture estimation. Therefore, prompt data can be constructed based on the first posture data of the target in the image to be annotated and the image information of the image to be annotated, and a first image containing the target can be generated based on the prompt data, thereby improving the defects of the posture information in the image to be annotated.

[0064] Exemplarily, the prompt data may be “target name: pedestrian; position information: center point coordinates: (300, 450) (this coordinate indicates the approximate center position of the target pedestrian in the image to be annotated); posture information (based on the first key point coordinates and confidence in the second posture data): head, coordinates: (290, 430), confidence: 0.98 (high confidence indicates that the key point is detected accurately); left shoulder, coordinates: (270, 480), confidence: 0.95; right shoulder, coordinates: (320, 480), confidence: 0.96; left elbow, coordinates: (250, 550), confidence: 0.85 (relatively low confidence, further verification may be required); right elbow, coordinates: (340, 540), confidence: 0.92; left wrist, coordinates: (240, 600), confidence: 0.88; right wrist, coordinates: (350, 590), confidence: 0.90; prompt information: use a red rectangle to mark the center position of the target pedestrian (290,450) and its surrounding area to intuitively display the target position; draw a human posture skeleton diagram based on the coordinates of each key point, connect the key points with line segments, and the color can be adjusted according to the confidence level (such as high confidence is green, medium is yellow, and low is red warning); mark the coordinates and confidence values ​​next to the key points for manual review and adjustment; remarks: please note that the left elbow key point has a low confidence level. It is recommended to review the image in this area or adjust the detection parameters to improve accuracy.

[0065] S41: Generate a first image including the target using a generative pre-trained model according to the prompt data.

[0066] In one embodiment of the present application, prompt data can be input into a generative pre-trained model, so as to reconstruct the posture corresponding to the target in the image to be annotated using the generative pre-trained model to obtain a first image. The generative pre-trained model can be a bidirectional encoder representation (Bidirectional Encoder Representations from Transformers, BERT) based on transfer learning, or a multimodal generative pre-trained model, which is not limited in the present application.

[0067] In this way, a first image with good quality can be generated according to the prompt data including the target posture and composition information, thereby avoiding the problem of low data annotation accuracy caused by low imaging quality of the image to be annotated.

[0068] like Figure 8 , is a flow chart of a method for determining depth information of a target provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining depth information of a target provided in an embodiment of the present application includes the following steps.

[0069] S50: predicting the depth value of each pixel in the image to be labeled based on the depth estimation model.

[0070] In one embodiment of the present application, the depth estimation model can be a convolutional neural network model, a graph neural network model, or a fully connected neural network model, which is not limited in the present application. Specifically, the depth estimation model can be used to extract features of the image to be annotated to obtain image features including visual information such as edges, textures, and colors. Through the multi-layer convolution and pooling operations in the depth estimation model, the semantic features of the image are gradually extracted. The depth value of each pixel in the image to be annotated is determined based on the motion information in the semantic features, the perspective relationship between the targets, the occlusion relationship, and other information.

[0071] S51: Determine a depth value of a first key point in the second posture data according to the depth value of each pixel.

[0072] In one embodiment of the present application, each pixel point in the image to be annotated can be traversed. When the coordinates of the traversed pixel point are the same as the coordinates of the first key point in any second posture data, the depth value of the pixel point can be determined to be the depth value of the first key point.

[0073] S52: Determine depth information of the target according to the depth value of the first key point.

[0074] In an embodiment of the present application, since the second posture data can be used to represent the posture information of the target in the first image, the depth information of the target can be determined using the depth value of each first key point in the second posture data.

[0075] In one embodiment of the present application, when the confidence of the first key point is greater than or equal to a preset confidence threshold, the first key point can be determined to be a visible key point; and the depth information of the target in the first image can be determined based on the average value of the depth values ​​of the visible key points. Specifically, when the confidence of the first key point reaches the confidence threshold, it indicates that the accuracy of the predicted first key point is high, so the probability of the first key point being blocked is low, and therefore the first key point can be determined to be a visible key point. For any target, the average of the depth values ​​of all visible key points in the second posture data of the target can be calculated, and the average value can be determined to be the depth information of the target.

[0076] In another embodiment of the present application, a weighted sum operation may be performed on the depth values ​​of the first key points based on the confidence of the first key points to obtain the depth information of the target in the first image. Specifically, for the second posture data of each target, the product of the depth value of each first key point and the confidence may be calculated, and the sum of the products corresponding to each first key point may be calculated to obtain the corresponding depth information.

[0077] like Fig. 9 , is a flow chart of a method for generating annotated images provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for generating annotated images provided in an embodiment of the present application includes the following steps.

[0078] S60: Determine a positional relationship between multiple targets according to the depth information.

[0079] In another embodiment of the present application, the depth information of the target can be mapped to three-dimensional coordinates or distances relative to the plane where the camera is located according to the camera coordinate system and the coordinates of the first key point in the first image. In this way, the relative positions of different targets can be compared according to the depth values ​​of the targets. The coordinate system in the three-dimensional space can also be used to represent the positions of the targets and calculate geometric relationships such as distances and angles between them.

[0080] S61, generating the annotated image according to the positional relationship between the multiple objects, the second posture data of the multiple objects, and the second image.

[0081] In another embodiment of the present application, the corresponding position of the target in the first image in the second image can be determined based on the positional relationship between multiple targets, and the image information corresponding to the target in the first image can be merged into the corresponding position in the second image. For example, when the coordinates of a pixel point belonging to a target in the first image are (450, 450), and the pixel value of the pixel point is 20, it can be determined that the pixel value of the pixel point with the coordinates of (450, 450) in the second image is 20. The correspondence between the second posture data and the pixel points corresponding to the target can also be determined, so as to annotate the posture of the target and obtain an annotated image containing the posture information of the target.

[0082] See also Fig.10 , Fig.10 9 is a functional module diagram of an image annotation device provided in an embodiment of the present application. An image annotation device 91 includes a posture estimation module 911, a generation module 912, a depth estimation module 913, and an annotation module 914. The module / unit referred to in the present application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and can complete fixed functions, which are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0083] The posture estimation module 911 is used to perform posture estimation processing on the image to be labeled to obtain first posture data of the target in the image to be labeled.

[0084] The posture estimation module 911 is further configured to determine second posture data from the first posture data.

[0085] The generating module 912 is used to generate a first image containing the target according to the second posture data and the image to be annotated.

[0086] The generating module 912 is further configured to generate a second image that does not include the target according to the image to be annotated; wherein the second image includes background information in the image to be annotated.

[0087] The depth estimation module 913 is used to perform depth estimation on the image to be labeled to obtain depth information of the target.

[0088] The annotation module 914 is configured to generate an annotated image according to the depth information, the first image, and the second image.

[0089] The posture estimation module 911 is also used to determine the score of the first posture data according to the coordinates of the first key point in the first posture data and the confidence of the first key point; when the score reaches a preset threshold, determine the corresponding first posture data as second posture data.

[0090] The generation module 912 is also used to determine prompt data based on the coordinates and confidence of the first key point in the second posture data, the image to be annotated and a pre-stored prompt template; wherein the prompt data is used to characterize the position information and posture information of the target in the image to be annotated; and based on the prompt data, a first image containing the target is generated using a generative pre-trained model.

[0091] The depth estimation module 913 is also used to predict the depth value of each pixel in the image to be annotated based on the depth estimation model; determine the depth value of the first key point in the second posture data according to the depth value of each pixel; and determine the depth information of the target according to the depth value of the first key point.

[0092] The depth estimation module 913 is further used to determine that the first key point is a visible key point when the confidence of the first key point is greater than or equal to a preset confidence threshold; and determine the depth information of the target in the first image based on the average value of the depth values ​​of the visible key points.

[0093] The depth estimation module 913 is further configured to perform a weighted sum operation on the depth values ​​of the first key points based on the confidence levels of the first key points to obtain depth information of the target in the first image.

[0094] The annotation module 914 is further configured to determine the positional relationship between the multiple targets according to the depth information; and generate the annotated image according to the positional relationship between the multiple targets, the second posture data of the multiple targets, and the second image.

[0095] See also Fig.11 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement an image annotation method described in any of the above embodiments.

[0096] In an embodiment of the present application, the electronic device 100 further includes a bus, and a computer program stored in the memory 12 and executable on the processor 13, such as an image annotation program.

[0097] Fig.11 Only the electronic device 100 having the memory 12 and the processor 13 is shown, and those skilled in the art can understand that Fig.11 The structure shown does not constitute a limitation on the electronic device 100 , and the electronic device 100 may include fewer or more components than shown in the figure, or combine some components, or arrange the components differently.

[0098] Combination Figure 2 The memory 12 in the electronic device 100 stores a plurality of computer-readable instructions to implement an image annotation method, and the processor 13 can execute the plurality of instructions to implement: performing posture estimation processing on the image to be annotated to obtain first posture data of the target in the image to be annotated; determining second posture data from the first posture data; generating a first image containing the target according to the second posture data and the image to be annotated; generating a second image not containing the target according to the image to be annotated; wherein the second image includes background information in the image to be annotated; performing depth estimation on the image to be annotated to obtain depth information of the target; generating an annotated image according to the depth information, the first image and the second image.

[0099] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 2 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0100] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the electronic device 100 may also include input and output devices, network access devices, etc.

[0101] It should be noted that the electronic device 100 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0102] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as a mobile hard disk of the electronic device 100. In other embodiments, the memory 12 can also be an external storage device of the electronic device 100, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device 100. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 100, such as a code of an image annotation program, etc., but also can be used to temporarily store data that has been output or is to be output.

[0103] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 100, and uses various interfaces and lines to connect various components of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing programs or modules stored in the memory 12 (for example, executing an image annotation program, etc.), and calling data stored in the memory 12.

[0104] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes the applications to implement the steps in each of the above-mentioned image annotation method embodiments, for example Figure 2 Steps shown.

[0105] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 100. For example, the computer program may be divided into a posture estimation module 911, a generation module 912, a depth estimation module 913, and a labeling module 914.

[0106] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of an image annotation method described in each embodiment of the present application.

[0107] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware devices to complete through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0108] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory, etc.

[0109] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0110] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Fig.11 Only one arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0111] An embodiment of the present application further provides a computer-readable storage medium (not shown), in which computer-readable storage medium is stored computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the image annotation method described in any of the above embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0113] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0115] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. An image annotation method, applied to electronic equipment, characterized in that: The method comprises: Performing posture estimation processing on the image to be labeled to obtain first posture data of the target in the image to be labeled; determining second posture data from the first posture data; Generating a first image including the target according to the second posture data and the image to be annotated; Generate a second image that does not contain the target according to the image to be annotated; wherein the second image includes background information in the image to be annotated; Performing depth estimation on the image to be labeled to obtain depth information of the target; A labeled image is generated according to the depth information, the first image, and the second image.

2. The image annotation method according to claim 1, characterized in that: Determining second posture data from the first posture data comprises: Determining a score of the first posture data according to the coordinates of the first key point in the first posture data and the confidence of the first key point; When the score reaches a preset threshold, the corresponding first posture data is determined to be second posture data.

3. The image annotation method according to claim 2, characterized in that: The step of generating a first image including the target according to the second posture data and the image to be annotated comprises: Determining prompt data according to the coordinates and confidence of the first key point in the second posture data, the image to be annotated, and a pre-stored prompt template; wherein the prompt data is used to characterize the position information and posture information of the target in the image to be annotated; According to the prompt data, a first image including the target is generated using a generative pre-trained model.

4. The image annotation method according to claim 1, characterized in that: The performing depth estimation on the image to be labeled to obtain depth information of the target includes: Predicting the depth value of each pixel in the image to be annotated based on the depth estimation model; Determine the depth value of the first key point in the second posture data according to the depth value of each pixel; Determine the depth information of the target according to the depth value of the first key point.

5. The image annotation method according to claim 4, characterized in that: Determining the depth information of the target according to the depth value of the first key point includes: When the confidence level of the first key point is greater than or equal to a preset confidence level threshold, determining that the first key point is a visible key point; Depth information of the object in the first image is determined according to an average value of the depth values ​​of the visible key points.

6. The image annotation method according to claim 4, characterized in that: Determining the depth information of the target according to the depth value of the first key point includes: Based on the confidence of the first key point, a weighted sum operation is performed on the depth value of the first key point to obtain depth information of the target in the first image.

7. The image annotation method according to claim 1, characterized in that: The generating the annotated image according to the depth information, the first image and the second image comprises: Determine the positional relationship between the multiple targets according to the depth information; The annotated image is generated according to the positional relationship between the multiple objects, the second posture data of the multiple objects, and the second image.

8. An image annotation device, characterized in that: The device comprises: A posture estimation module, used to perform posture estimation processing on the image to be annotated to obtain first posture data of the target in the image to be annotated; The posture estimation module is further used to determine second posture data from the first posture data; A generating module, used for generating a first image including the target according to the second posture data and the image to be annotated; The generating module is further configured to generate a second image that does not contain the target according to the image to be annotated; wherein the second image includes background information in the image to be annotated; A depth estimation module, used to perform depth estimation on the image to be labeled to obtain depth information of the target; A labeling module is used to generate a labeled image according to the depth information, the first image and the second image.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, and the processor is used to implement the image annotation method according to any one of claims 1 to 7 when executing a computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image annotation method according to any one of claims 1 to 7 is implemented.

Images