Method, apparatus and image processing system for image tagging

Abstract

The application provides a method, device and image processing system for image labeling, the method comprising: obtaining a target image frame, the target image frame being an image frame in which a target object is partially occluded in video data; determining a first image frame and a second image frame according to the target image frame, the first image frame being an image frame in which the target object is not occluded before the target image frame, and the second image frame being an image frame in which the target object is not occluded after the target image frame; determining a target point cloud according to a first point cloud and a second point cloud, the first point cloud being a point cloud corresponding to the target object in the first image frame, the second point cloud being a point cloud corresponding to the target object in the second image frame, and the target point cloud being a point cloud corresponding to the target object in the target image frame; projecting the target point cloud to the target image frame and labeling, to obtain a labeled target image frame, the labeled target image frame including the target object labeled, and solving the problem of incomplete labeling of the partially occluded target object in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a method and device for image labeling, a computer readable storage medium, a processor and an image processing system. BACKGROUND

[0002] The output of the current 2d image recognition network is 2d bbox class confidence and the like, and if the subsequent tracking and distance measurement method depends on the size and position of the 2d bbox frame, stable and accurate bbox is a prerequisite. At present, when the target object is blocked and the like, the bbox may become smaller, disappear, and the center point may be shifted, which affects the stability and accuracy of the subsequent program.

[0003] The above information disclosed in the background section is only used to enhance the understanding of the background of the technology described herein, therefore, the background section can include some information which does not form the prior art known in the country to those skilled in the art. SUMMARY

[0004] The main purpose of the present application is to provide a method and device for image labeling, a computer readable storage medium, a processor and an image processing system, so as to solve the problem that the label of the partially blocked target object in the prior art is not complete.

[0005] According to an aspect of an embodiment of the present application, a method for image labeling is provided, comprising: acquiring a target image frame, the target image frame being an image frame in which a target object is partially blocked in video data; determining a first image frame and a second image frame according to the target image frame, the first image frame being an image frame in which the target object is not blocked before the target image frame, and the second image frame being an image frame in which the target object is not blocked after the target image frame; determining a target point cloud according to a first point cloud and a second point cloud, the first point cloud being a point cloud corresponding to the target object in the first image frame, the second point cloud being a point cloud corresponding to the target object in the second image frame, and the target point cloud being a point cloud corresponding to the target object in the target image frame; projecting the target point cloud to the target image frame and labeling, to obtain a labeled target image frame, the labeled target image frame including the target object labeled.

[0006] Optionally, projecting the target point cloud to the target image frame and labeling to obtain a labeled target image frame comprises: projecting the target point cloud to the target image frame and labeling to obtain a first bounding box; identifying and labeling the target object in the target image frame to obtain a second bounding box; in the case that the coincidence degree of the first bounding box and the second bounding box is greater than a predetermined value, taking the first bounding box as the label of the target object to form the labeled target image frame.

[0007] Optionally, projecting the target point cloud to the target image frame and labeling to obtain a first bounding box comprises: performing coordinate system conversion on the points of the target point cloud, so that the target point cloud is projected to the target image frame to obtain a projection region; labeling the projection region to obtain the first bounding box.

[0008] Optionally, before the case that the coincidence degree of the first bounding box and the second bounding box is greater than a predetermined value, taking the first bounding box as the label of the target object to form the labeled target image frame, the method further comprises: calculating the overlapping area of the internal region of the first bounding box and the internal region of the second bounding box to obtain a first area; calculating the area of the internal region of the second bounding box to obtain a second area; calculating the ratio of the first area and the second area to obtain the coincidence degree.

[0009] Optionally, determining a target point cloud according to a first point cloud and a second point cloud comprises: matching the points of the first point cloud and the points of the second point cloud one by one to obtain a plurality of point pairs; determining the three-dimensional coordinates of a plurality of target points according to the three-dimensional coordinates of each point pair to obtain the target point cloud, the target points corresponding one by one to the point pairs.

[0010] Optionally, determining the three-dimensional coordinates of a plurality of target points according to the three-dimensional coordinates of each point pair to obtain the target point cloud comprises: a first calculation step, calculating a moving speed according to the three-dimensional coordinates of one point pair, the timestamp of the first image frame and the timestamp of the second image frame; a second calculation step, calculating a moving distance according to the moving speed, the timestamp of the target image frame and the timestamp of the first image frame; a determination step, determining the three-dimensional coordinates of the target point according to the moving distance and the three-dimensional coordinates of the point pair; sequentially repeating the first calculation step, the second calculation step and the determination step at least once until the three-dimensional coordinates of all target points are obtained, and the target point cloud is obtained.

[0011] According to a further aspect of the embodiments of the present application, there is also provided an apparatus for image labeling, comprising: an obtaining unit configured to obtain a target image frame, the target image frame being an image frame in which a target object is occluded in video data; a first determining unit configured to determine a first image frame and a second image frame according to the target image frame, the first image frame being an image frame in which the target object is not occluded before the target image frame, and the second image frame being an image frame in which the target object is not occluded after the target image frame; a second determining unit configured to determine a target point cloud according to a first point cloud and a second point cloud, the first point cloud being a point cloud corresponding to the target object in the first image frame, the second point cloud being a point cloud corresponding to the target object in the second image frame, and the target point cloud being a point cloud corresponding to the target object in the target image frame; and a projecting unit configured to project the target point cloud to the target image frame and label the target image frame to obtain a labeled target image frame, the labeled target image frame comprising the target object labeled.

[0012] According to a further aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, the computer-readable storage medium comprising a stored program, wherein the program performs any of the methods.

[0013] According to a further aspect of the embodiments of the present application, there is also provided a processor configured to execute a program, wherein the program performs any of the methods.

[0014] According to a further aspect of the embodiments of the present application, there is also provided an image processing system, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for performing any of the methods.

[0015] In the embodiment of the present application, in the image labeling method, first, a target image frame is obtained, the target image frame being an image frame in which a target object in video data is partially occluded; then, a first image frame and a second image frame are determined according to the target image frame, the first image frame being an image frame in which the target object is not occluded before the target image frame, and the second image frame being an image frame in which the target object is not occluded after the target image frame; then, a target point cloud is determined according to a first point cloud and a second point cloud, the first point cloud being a point cloud corresponding to the target object in the first image frame, the second point cloud being a point cloud corresponding to the target object in the second image frame, and the target point cloud being a point cloud corresponding to the target object in the target image frame; finally, the target point cloud is projected to the target image frame and labeled to obtain a labeled target image frame, the labeled target image frame including the target object labeled. This method finds an image frame in which the target object is partially occluded as a target image frame, finds image frames in which the target object is not occluded before and after the target image frame, determines a point cloud corresponding to the target object in the target image frame according to the trajectory prediction of the point clouds of the target object in the two image frames, projects the point cloud to the target image frame and labels it, and obtains a labeled target image frame, thereby avoiding directly labeling a partially occluded target object to cause incomplete labeling, solving the problem of incomplete labeling of a partially occluded target object in the prior art, ensuring the accuracy and stability of the labeling of the target object, and facilitating subsequent ranging, tracking, and training. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application and are incorporated herein for a purpose of explanations and are not intended to limit the present application. In the drawings:

[0017] Figure 1 A flowchart of an image labeling method according to an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of an image labeling device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0020] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0021] It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements can also be present. In addition, when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can be present.

[0022] As mentioned in the background, the existing technology is not complete for labeling the partially occluded target object, in order to solve the above problems, in a typical embodiment of the present application, a method and device for image labeling, computer readable storage medium, processor and image processing system are provided.

[0023] According to the embodiments of the present application, a method for image labeling is provided.

[0024] Figure 1 is a flowchart of the method for image labeling according to the embodiments of the present application. As shown in Figure 1 the method comprises the following steps:

[0025] Step S101, obtaining a target image frame, the target image frame being an image frame in which a target object is partially occluded in video data;

[0026] In the obstacle tracking process, the target object is occluded, which can cause the bounding box bbox to become smaller, disappear, the center point to shift, and other problems, affecting the stability and accuracy of subsequent tracking. In order to solve the above problems, first, an image frame in which the target object is partially occluded in the video data is obtained as a target image frame, and then the problems of the bounding box bbox becoming smaller, disappearing, the center point shifting, and other problems in the target image frame are solved.

[0027] It should be noted that the above-mentioned target image frame is an image frame in which the target object is partially occluded, not an image frame in which the target object is completely occluded. The image frame in which the target object is completely occluded cannot generate partial bounding boxes bbox, so that the partial bounding boxes bbox cannot be compared with the complete bounding box bbox generated according to the positioning, resulting in inaccurate complete labeling frame.

[0028] Step S102: Determine a first image frame and a second image frame based on the target image frame. The first image frame is an image frame before the target image frame in which the target object is not obscured, and the second image frame is an image frame after the target image frame in which the target object is not obscured.

[0029] Typically, the target object goes from being unoccluded to being occluded and then back to being unoccluded, forming a continuous tracking process. Therefore, it is possible to find the image frames before the target image frame where the target object is not occluded and the image frames after the target image frame where the target object is not occluded, namely the first image frame and the second image frame.

[0030] Of course, the first image frame mentioned above may not exist. That is, before the target object is occluded, no tracking is performed, and it is not necessary to obtain the complete bounding box (bbox) of the target object in the target image frame to ensure the stability of tracking. Tracking can start from the second image frame mentioned above. Similarly, the second image frame mentioned above may not exist. That is, after the target object is occluded, no tracking is performed, and it is not necessary to obtain the complete bounding box (bbox) of the target object in the target image frame.

[0031] Step S103: Determine the target point cloud based on the first point cloud and the second point cloud. The first point cloud is the point cloud corresponding to the target object in the first image frame, the second point cloud is the point cloud corresponding to the target object in the second image frame, and the target point cloud is the point cloud corresponding to the target object in the target image frame.

[0032] Optionally, the present invention does not limit the specific process of determining the target point cloud based on the first point cloud and the second point cloud, and any feasible method is within the protection scope of the present invention.

[0033] For example, in an optional implementation, step S103 above includes:

[0034] Step S1031: Match the points of the first point cloud and the points of the second point cloud one by one to obtain multiple point pairs.

[0035] Step S1032: Determine the three-dimensional coordinates of multiple target points based on the three-dimensional coordinates of each of the above point pairs to obtain the above target point cloud, wherein each of the above target points corresponds one-to-one with the above point pairs.

[0036] In the above embodiment, by matching the first point cloud and the second point cloud, the same point on the target object is found in the corresponding point of the first point cloud and the corresponding point of the second point cloud, a point pair is formed, that is, one point of the target object corresponds to one point, and according to the point pair, the three-dimensional coordinates of the point of the target object at the time corresponding to the target image frame during the movement of the target object from the position of the first point cloud to the position of the second point cloud are determined, that is, the three-dimensional coordinates of the target point, one point of the target object corresponds to one target point, and a plurality of points of the target object correspond to a plurality of target points, and the plurality of target points form the target point cloud. In this embodiment, the calculation is performed by one-to-one correspondence of the point cloud, and the deformation of the target point cloud is avoided to ensure the accuracy of the bounding box.

[0037] Optionally, the present application does not limit the specific process of determining the three-dimensional coordinates of a plurality of target points according to the three-dimensional coordinates of each point pair to obtain the target point cloud, and any feasible manner belongs to the protection scope of the present application.

[0038] For example, in an optional embodiment, the step S1032 comprises:

[0039] Step S10321, a first calculation step, calculating a moving speed according to the three-dimensional coordinates of a point pair, the timestamp of the first image frame, and the timestamp of the second image frame;

[0040] Step S10322, a second calculation step, calculating a moving distance according to the moving speed, the timestamp of the target image frame, and the timestamp of the first image frame;

[0041] Step S10323, a determination step, determining the three-dimensional coordinates of the target point according to the moving distance and the three-dimensional coordinates of the point pair;

[0042] Step S10324, sequentially repeating the first calculation step, the second calculation step, and the determination step at least once until the three-dimensional coordinates of all target points are obtained, and the target point cloud is obtained.

[0043] In the above embodiment, the target object is assumed to move at a constant speed in a straight line from the position of the first point cloud to the position of the second point cloud, that is, the moving speed can be calculated according to the timestamp and the displacement, and the displacement from the position of the first point cloud to the target point cloud can be calculated, so that the three-dimensional coordinates of the target point cloud can be obtained according to the displacement and the three-dimensional coordinates of the first point cloud. In this embodiment, by calculating the moving distance from the first point cloud to the target point cloud, the first point cloud is translated along the movement direction by the moving distance, and the target point cloud is obtained, which ensures that the point cloud will not be deformed and further ensures the accuracy of the bounding box.

[0044] Step S104, projecting the target point cloud to the target image frame and labeling to obtain a labeled target image frame, wherein the labeled target image frame comprises the target object labeled.

[0045] Optionally, the present application does not limit the specific process of projecting the target point cloud to the target image frame and labeling to obtain a labeled target image frame, and any feasible manner belongs to the protection scope of the present application.

[0046] For example, in an optional embodiment, step S104 comprises:

[0047] Step S1041, projecting the target point cloud to the target image frame and labeling to obtain a first bounding box;

[0048] Step S1042, identifying and labeling the target object in the target image frame to obtain a second bounding box;

[0049] Step S1043, in the case where the coincidence degree of the first bounding box and the second bounding box is greater than a predetermined value, taking the first bounding box as the label of the target object to form the labeled target image frame.

[0050] In the above embodiment, the target point cloud is projected to the target image frame through the conversion from three dimensions to two dimensions to form a complete image of the target object, and then labeling is performed, so that a complete bounding box of the target object, i.e., the first bounding box, is obtained. In addition, the target object in the target image frame is directly identified and labeled, and since the target object is partially blocked, only the bounding box of the unblocked part, i.e., the second bounding box, can be obtained. In the case where the coincidence degree of the first bounding box and the second bounding box is greater than a predetermined value, it can be determined that the first bounding box is the bounding box of the target object, i.e., the first bounding box is taken as the label of the target object to form the labeled target image frame. In this embodiment, the partial bounding box bbox obtained by identification is compared with the complete bounding box bbox projected according to the target point cloud, and only when the coincidence degree is greater than a predetermined value, it can be determined that the complete bounding box bbox is qualified, thereby further ensuring the accuracy of the bounding box bbox.

[0051] In addition, when the coincidence degree is less than or equal to a predetermined value, the first image frame and the second image frame can be replaced until the first bounding box with a coincidence degree meeting the requirement is found.

[0052] Optionally, the present application does not limit the specific process of projecting the target point cloud to the target image frame and labeling to obtain a first bounding box, and any feasible manner belongs to the protection scope of the present application.

[0053] For example, in an alternative embodiment, the step S1041 comprises:

[0054] Step S10411, coordinate system conversion is performed on the points of the target point cloud, so that the target point cloud is projected onto the target image frame to obtain a projection region.

[0055] Step S10412, the projection region is labeled to obtain the first bounding box.

[0056] In the above embodiment, the points of the target point cloud are converted from the world coordinate system to the two-dimensional coordinate system of the target image frame, i.e., the target point cloud is projected onto the target image frame to obtain a projection region, and the projection region is labeled to obtain the bounding box of the target object, i.e., the first bounding box. This embodiment simplifies the complex projection into a simple calculation process by coordinate system conversion to project the target point cloud onto the target image frame, thereby improving the efficiency.

[0057] Moreover, the process of identifying and labeling the target object in the target image frame to obtain the second bounding box is consistent with the principle of obtaining the first bounding box, which will not be described here.

[0058] In addition, in an alternative embodiment, before the step S1043, the method further comprises:

[0059] Step 1.1, calculating the overlapping area of the internal region of the first bounding box and the internal region of the second bounding box to obtain a first area;

[0060] Step 1.2, calculating the area of the internal region of the second bounding box to obtain a second area;

[0061] Step 1.3, calculating the ratio of the first area and the second area to obtain the coincidence degree.

[0062] In the above embodiment, the internal region of the first bounding box and the internal region of the second bounding box partially overlap, and the proportion of the overlapping region in the internal region of the second bounding box is calculated, i.e., the ratio of the area of the overlapping region to the area of the internal region of the second bounding box, to obtain the coincidence degree. The greater the proportion of the overlapping region, the higher the coincidence degree. This embodiment calculates the coincidence degree by area, and other suitable methods can also be selected to calculate the coincidence degree to ensure the accuracy of the bounding box bbox.

[0063] In the method for image labeling, first, a target image frame is obtained, the target image frame being an image frame in which a target object in video data is partially occluded; then, a first image frame and a second image frame are determined according to the target image frame, the first image frame being an image frame in which the target object is not occluded before the target image frame, and the second image frame being an image frame in which the target object is not occluded after the target image frame; subsequently, a target point cloud is determined according to a first point cloud and a second point cloud, the first point cloud being a point cloud corresponding to the target object in the first image frame, the second point cloud being a point cloud corresponding to the target object in the second image frame, and the target point cloud being a point cloud corresponding to the target object in the target image frame; finally, the target point cloud is projected to the target image frame and labeled, to obtain a labeled target image frame, the labeled target image frame including the target object labeled. The method finds an image frame in which the target object is partially occluded as the target image frame, finds image frames in which the target object is not occluded before and after the target image frame, determines the point cloud corresponding to the target object in the target image frame according to the point clouds of the target object in the two image frames, projects the point cloud to the target image frame and labels the target image frame, to obtain a labeled target image frame, thereby avoiding directly labeling the target object partially occluded, solving the problem that the label of the target object partially occluded is incomplete in the prior art, ensuring the accuracy and stability of the label of the target object, and facilitating subsequent ranging, tracking and training.

[0064] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0065] The embodiment of the present application also provides an image labeling device. It should be noted that the image labeling device of the embodiment of the present application can be used to execute the method for image labeling provided by the embodiment of the present application. The image labeling device provided by the embodiment of the present application is introduced as follows.

[0066] Figure 2 is a schematic diagram of the image labeling device according to the embodiment of the present application. As shown in Figure 2 , the device includes:

[0067] The acquisition unit 10 is configured to acquire a target image frame, the target image frame being an image frame in which a target object in video data is partially occluded.

[0068] In the obstacle tracking process, the target object is blocked, which can cause the bounding box bbox to become smaller, disappear, and the center point to shift, affecting the stability and accuracy of subsequent tracking. In order to solve the above problems, first, the image frame in which the target object is partially blocked in the video data is obtained as the target image frame, and then the problem of the bounding box bbox becoming smaller, disappearing, and the center point shifting in the target image frame is solved.

[0069] It should be noted that the above target image frame is an image frame in which the target object is partially blocked, not an image frame in which the target object is completely blocked. The image frame in which the target object is completely blocked cannot generate partial bounding boxes bbox, so it cannot compare the partial bounding boxes bbox with the complete bounding box bbox generated according to the positioning, resulting in inaccurate complete bounding boxes.

[0070] The first determination unit 20 is configured to determine a first image frame and a second image frame according to the target image frame. The first image frame is an image frame in which the target object is not blocked before the target image frame. The second image frame is an image frame in which the target object is not blocked after the target image frame.

[0071] Generally, the target object is never blocked to be blocked and then unblocked, forming a continuous tracking process. Therefore, the image frame in which the target object is not blocked before the target image frame and the image frame in which the target object is not blocked after the target image frame, i.e., the first image frame and the second image frame, can be found.

[0072] Of course, the first image frame can not exist, i.e., the target object is not tracked before being blocked, and the complete bounding box bbox of the target object of the target image frame does not need to be obtained to ensure the stability of the tracking. The tracking can start from the second image frame. Similarly, the second image frame can not exist, i.e., the target object is not tracked after being blocked, and the complete bounding box bbox of the target object of the target image frame does not need to be obtained.

[0073] The second determination unit 30 is configured to determine a target point cloud according to a first point cloud and a second point cloud. The first point cloud is a point cloud corresponding to the target object in the first image frame. The second point cloud is a point cloud corresponding to the target object in the second image frame. The target point cloud is a point cloud corresponding to the target object in the target image frame.

[0074] Optionally, the present application does not limit the specific process of determining the target point cloud according to the first point cloud and the second point cloud. Any feasible way belongs to the protection scope of the present application.

[0075] For example, in an optional embodiment, the second determination unit comprises:

[0076] a matching module configured to match the points of the first point cloud and the points of the second point cloud one by one to obtain a plurality of point pairs;

[0077] a determining module configured to determine the three-dimensional coordinates of a plurality of target points according to the three-dimensional coordinates of the point pairs to obtain the target point cloud, the target points corresponding to the point pairs one by one.

[0078] In the embodiment, by matching the first point cloud and the second point cloud, the same point on the target object is found in the corresponding point in the first point cloud and the corresponding point in the second point cloud, and a point pair is formed, that is, one point of the target object corresponds to one point pair. According to the point pair, the three-dimensional coordinates of the point of the target object at the time corresponding to the target image frame during the movement of the target object from the position of the first point cloud to the position of the second point cloud are determined, that is, the three-dimensional coordinates of the target point. One point of the target object corresponds to one target point, and a plurality of points of the target object correspond to a plurality of target points. The plurality of target points form the target point cloud. In this embodiment, the calculation is performed by one-to-one correspondence of the point cloud, and the deformation of the target point cloud is avoided to ensure the accuracy of the bounding box.

[0079] Optionally, the specific process of determining the three-dimensional coordinates of a plurality of target points according to the three-dimensional coordinates of the point pairs to obtain the target point cloud is not limited, and any feasible manner belongs to the protection scope of the present application.

[0080] For example, in an optional embodiment, the determining module comprises:

[0081] a first calculation sub-module configured to perform a first calculation step, and calculate a moving speed according to the three-dimensional coordinates of a point pair, the timestamp of the first image frame, and the timestamp of the second image frame;

[0082] a second calculation sub-module configured to perform a second calculation step, and calculate a moving distance according to the moving speed, the timestamp of the target image frame, and the timestamp of the first image frame;

[0083] a determining sub-module configured to perform a determining step, and determine the three-dimensional coordinates of a target point according to the moving distance and the three-dimensional coordinates of the point pair;

[0084] a repeating sub-module configured to repeat the first calculation step, the second calculation step, and the determining step at least once in sequence until the three-dimensional coordinates of all the target points are obtained, and the target point cloud is obtained.

[0085] In the above embodiment, the default target object moves from the first point cloud position to the second point cloud position at a constant speed in a straight line, the moving speed is calculated according to the time stamp and the displacement, the displacement from the first point cloud position to the target point cloud is calculated, and the three-dimensional coordinates of the target point cloud are obtained according to the displacement and the three-dimensional coordinates of the first point cloud. In this embodiment, the moving distance from the first point cloud to the target point cloud is calculated, and the first point cloud is translated along the moving direction by the moving distance, so that the target point cloud is obtained, which ensures that the point cloud will not be deformed and further ensures the accuracy of the bounding box.

[0086] The projection unit 40 is configured to project the target point cloud to the target image frame and label the target image frame to obtain a labeled target image frame, wherein the labeled target image frame comprises the target object labeled.

[0087] Optionally, the present application does not limit the specific process of projecting the target point cloud to the target image frame and labeling the target image frame to obtain a labeled target image frame, and any feasible manner belongs to the protection scope of the present application.

[0088] For example, in an optional embodiment, the projection unit comprises:

[0089] The first processing module is configured to project the target point cloud to the target image frame and label the target image frame to obtain a first bounding box.

[0090] The second processing module is configured to identify and label the target object in the target image frame to obtain a second bounding box.

[0091] The third processing module is configured to, in the case that the coincidence degree of the first bounding box and the second bounding box is greater than a predetermined value, take the first bounding box as the label of the target object to form the labeled target image frame.

[0092] In the above embodiment, the target point cloud is projected to the target image frame by conversion from three-dimensional to two-dimensional to form a complete image of the target object, and then labeled, so that a complete bounding box of the target object, i.e. the first bounding box, is obtained. In addition, the target object in the target image frame is directly identified and labeled, and since the target object is partially blocked, only the bounding box of the unblocked part, i.e. the second bounding box, can be obtained. In the case that the coincidence degree of the first bounding box and the second bounding box is greater than a predetermined value, it can be determined that the first bounding box is the bounding box of the target object, i.e. the first bounding box is taken as the label of the target object to form the labeled target image frame. In this embodiment, the partial bounding box bbox obtained by identification is compared with the complete bounding box bbox projected according to the target point cloud, and only when the coincidence degree is greater than a predetermined value, it can be determined that the complete bounding box bbox is qualified, which further ensures the accuracy of the bounding box bbox.

[0093] In addition, when the coincidence degree is less than or equal to a predetermined value, the first image frame and the second image frame can be replaced until the first bounding box with a coincidence degree meeting the requirement is found.

[0094] Optionally, the present application does not limit the specific process of projecting the target point cloud to the target image frame and labeling to obtain the first bounding box, and any feasible manner belongs to the protection scope of the present application.

[0095] For example, in an optional embodiment, the first processing module comprises:

[0096] a conversion submodule, configured to perform coordinate system conversion on the points of the target point cloud, so that the target point cloud is projected to the target image frame to obtain a projection region;

[0097] a processing submodule, configured to label the projection region to obtain the first bounding box.

[0098] In the above embodiment, the points of the target point cloud are converted from the world coordinate system to the two-dimensional coordinate system of the target image frame, so that the target point cloud is projected to the target image frame to obtain a projection region, and the projection region is labeled to obtain the bounding box of the target object, i.e., the first bounding box. This embodiment simplifies the complex projection to a simple calculation process by converting the coordinate system to project the target point cloud onto the target image frame, thereby improving the efficiency.

[0099] In addition, the process of identifying and labeling the target object in the target image frame to obtain the second bounding box is consistent with the principle of obtaining the first bounding box, which will not be repeated here.

[0100] In addition, in an optional embodiment, the projection unit further comprises:

[0101] a first calculation module, configured to calculate the overlapping area of the internal region of the first bounding box and the internal region of the second bounding box to obtain a first area;

[0102] a second calculation module, configured to calculate the area of the internal region of the second bounding box to obtain a second area;

[0103] a third calculation module, configured to calculate the ratio of the first area and the second area to obtain the coincidence degree.

[0104] In the above embodiment, the internal region of the first bounding box and the internal region of the second bounding box partially overlap, the proportion of the overlapping region in the internal region of the second bounding box is calculated, that is, the ratio of the area of the overlapping region to the area of the internal region of the second bounding box, so that the overlap degree is obtained. The greater the proportion of the overlapping region, the higher the overlap degree. In this embodiment, the overlap degree is calculated by area. Other suitable methods can also be selected to calculate the overlap degree to ensure the accuracy of the bounding box bbox.

[0105] In the above image labeling device, the acquisition unit acquires a target image frame, the target image frame being an image frame in which a target object in video data is partially occluded; the first determination unit determines a first image frame and a second image frame according to the target image frame, the first image frame being an image frame in which the target object is not occluded before the target image frame, and the second image frame being an image frame in which the target object is not occluded after the target image frame; the second determination unit determines a target point cloud according to a first point cloud and a second point cloud, the first point cloud being a point cloud corresponding to the target object in the first image frame, the second point cloud being a point cloud corresponding to the target object in the second image frame, and the target point cloud being a point cloud corresponding to the target object in the target image frame; and the projection unit projects the target point cloud to the target image frame and labels it to obtain a labeled target image frame, the labeled target image frame including the target object labeled. The device finds an image frame in which the target object is partially occluded as a target image frame, and finds image frames in which the target object is not occluded before and after the target image frame, so as to determine the point cloud corresponding to the target object in the target image frame according to the point clouds of the target objects in the two image frames, project the obtained point cloud to the target image frame and label it to obtain a labeled target image frame, thereby avoiding incomplete labeling of the partially occluded target object and solving the problem of incomplete labeling of the partially occluded target object in the prior art, ensuring the accuracy and stability of the labeling of the target object and facilitating subsequent ranging, tracking and training.

[0106] The image labeling device includes a processor and a memory, and the acquisition unit, the first determination unit, the second determination unit and the projection unit are stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0107] The processor includes a core, and the core retrieves the corresponding program units from the memory. The core can be set to one or more, and the problem of incomplete labeling of the partially occluded target object in the prior art can be solved by adjusting the core parameters.

[0108] The memory can include non-persistent memory in a computer readable medium, random access memory (RAM) and / or non-volatile memory, such as read only memory (ROM) or flash memory, including at least one memory chip.

[0109] The embodiment of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the method.

[0110] The embodiment of the present application provides a processor, which is used for running a program, and the program is executed to realize the method.

[0111] The embodiment of the present application provides an image processing system, and the device comprises a processor, a memory and a program stored in the memory and capable of running on the processor, and the processor realizes at least the following steps when executing the program.

[0112] Step S101, acquiring a target image frame, the target image frame being an image frame in which a target object in video data is partially occluded;

[0113] Step S102, determining a first image frame and a second image frame according to the target image frame, the first image frame being an image frame in which the target object is not occluded before the target image frame, and the second image frame being an image frame in which the target object is not occluded after the target image frame;

[0114] Step S103, determining a target point cloud according to a first point cloud and a second point cloud, the first point cloud being a point cloud corresponding to the target object in the first image frame, the second point cloud being a point cloud corresponding to the target object in the second image frame, and the target point cloud being a point cloud corresponding to the target object in the target image frame;

[0115] Step S104, projecting the target point cloud to the target image frame and labeling to obtain a labeled target image frame, and the labeled target image frame comprises the target object labeled.

[0116] The image processing system in the present application can be a server, a PC, a PAD, a mobile phone or the like.

[0117] The present application further provides a computer program product, which is adapted to execute the program initialized with at least the following method steps when executed on a data processing device:

[0118] Step S101, acquiring a target image frame, the target image frame being an image frame in which a target object in video data is partially occluded;

[0119] Step S102, determining a first image frame and a second image frame according to the target image frame, the first image frame being an image frame in which the target object is not blocked before the target image frame, and the second image frame being an image frame in which the target object is not blocked after the target image frame;

[0120] Step S103, determining a target point cloud according to a first point cloud and a second point cloud, the first point cloud being a point cloud corresponding to the target object in the first image frame, the second point cloud being a point cloud corresponding to the target object in the second image frame, and the target point cloud being a point cloud corresponding to the target object in the target image frame;

[0121] Step S104, projecting the target point cloud to the target image frame and labeling to obtain a labeled target image frame, the labeled target image frame including the target object labeled.

[0122] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0123] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-described device embodiments are only schematic, for example, the division of the above-mentioned units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0124] The units described as separate components above can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0125] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0126] The integrated unit described above, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a computer-readable storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the above-mentioned method of each embodiment of the present application. The aforementioned computer-readable storage medium includes: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0127] From the above description, it can be seen that the above-mentioned embodiments of the present application achieve the following technical effects:

[0128] 1) In the image labeling method of the present application, first, a target image frame is obtained, the target image frame being an image frame in which a target object in video data is partially occluded; then, a first image frame and a second image frame are determined according to the target image frame, the first image frame being an image frame in which the target object is not occluded before the target image frame, and the second image frame being an image frame in which the target object is not occluded after the target image frame; thereafter, a target point cloud is determined according to a first point cloud and a second point cloud, the first point cloud being a point cloud corresponding to the target object in the first image frame, the second point cloud being a point cloud corresponding to the target object in the second image frame, and the target point cloud being a point cloud corresponding to the target object in the target image frame; finally, the target point cloud is projected to the target image frame and labeled to obtain a labeled target image frame, the labeled target image frame including the target object labeled. This method finds an image frame in which the target object is partially occluded as a target image frame, finds image frames in which the target object is not occluded before and after the target image frame, determines the point cloud corresponding to the target object in the target image frame according to the point clouds of the target object corresponding to the two image frames, projects the obtained point cloud to the target image frame and labels it, and obtains a labeled target image frame, which can avoid directly labeling a partially occluded target object and lead to incomplete labeling, solve the problem of incomplete labeling of a partially occluded target object in the prior art, ensure the accuracy and stability of the labeling of the target object, and facilitate subsequent ranging, tracking and training.

[0129] 2)、the image labeling device of the application, the acquisition unit acquires the target image frame, the target image frame is the image frame in which the target object part is blocked in the video data;The first determination unit determines the first image frame and the second image frame according to the target image frame, the first image frame is the image frame in which the target object is not blocked before the target image frame, and the second image frame is the image frame in which the target object is not blocked after the target image frame;The second determination unit determines the target point cloud according to the first point cloud and the second point cloud, the first point cloud is the point cloud corresponding to the target object in the first image frame, the second point cloud is the point cloud corresponding to the target object in the second image frame, and the target point cloud is the point cloud corresponding to the target object in the target image frame;The projection unit projects the target point cloud to the target image frame and labels, and obtains the labeled target image frame, and the labeled target image frame includes the target object labeled. The device finds the image frame in which the target object part is blocked as the target image frame, and the image frames in which the target object is not blocked before and after the target image frame, so as to determine the point cloud corresponding to the target object in the target image frame according to the point clouds of the target objects corresponding to the two, and project the obtained point cloud to the target image frame and label, and obtain the labeled target image frame, which can avoid directly labeling the partially blocked target object, solve the problem that the label of the partially blocked target object is incomplete in the prior art, ensure the accuracy and stability of the label of the target object, and facilitate subsequent ranging, tracking and training.

[0130] The above only describes the preferred embodiments of the application and is not used to limit the application. Those skilled in the art can make various modifications and changes to the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A method for labeling images, characterized in that, include: Acquire a target image frame, wherein the target image frame is an image frame in the video data in which part of the target object is occluded; A first image frame and a second image frame are determined based on the target image frame. The first image frame is an image frame before the target image frame in which the target object is not occluded, and the second image frame is an image frame after the target image frame in which the target object is not occluded. The target point cloud is determined based on the first point cloud and the second point cloud. The first point cloud is the point cloud corresponding to the target object in the first image frame, and the second point cloud is the point cloud corresponding to the target object in the second image frame. The target point cloud is the point cloud corresponding to the target object in the target image frame. The target point cloud is projected onto the target image frame and labeled to obtain a labeled target image frame, wherein the labeled target image frame includes the labeled target object; Determining the target point cloud based on the first point cloud and the second point cloud includes: matching the points of the first point cloud and the points of the second point cloud one-to-one to obtain multiple point pairs; During the process of determining the position of the target object from the position of the first point cloud to the position of the second point cloud based on the point pairs, the three-dimensional coordinates of multiple target points are determined based on the three-dimensional coordinates of each point pair to obtain the target point cloud, and the target points correspond one-to-one with the point pairs.

2. The method according to claim 1, characterized in that, Projecting the target point cloud onto the target image frame and labeling it yields a labeled target image frame, including: The target point cloud is projected onto the target image frame and labeled to obtain the first bounding box; The target object in the target image frame is identified and labeled to obtain a second bounding box; If the overlap between the first bounding box and the second bounding box is greater than a predetermined value, the first bounding box is used as the label of the target object to form the label target image frame.

3. The method according to claim 2, characterized in that, Projecting the target point cloud onto the target image frame and labeling it yields a first bounding box, including: The points of the target point cloud are transformed into a coordinate system so that the target point cloud is projected onto the target image frame to obtain the projection area; The projected area is labeled to obtain the first bounding box.

4. The method according to claim 2, characterized in that, Before forming the labeled target image frame by using the first bounding box as a label for the target object when the overlap between the first bounding box and the second bounding box is greater than a predetermined value, the method further includes: Calculate the overlapping area of ​​the inner regions of the first bounding box and the second bounding box to obtain the first area; Calculate the area of ​​the interior region of the second bounding box to obtain the second area; The overlap ratio is obtained by calculating the ratio of the first area to the second area.

5. The method according to claim 1, characterized in that, The three-dimensional coordinates of multiple target points are determined based on the three-dimensional coordinates of each pair of points to obtain the target point cloud, including: The first calculation step involves calculating the moving speed based on the three-dimensional coordinates of a pair of points, the timestamp of the first image frame, and the timestamp of the second image frame. The second calculation step involves calculating the moving distance based on the moving speed, the timestamp of the target image frame, and the timestamp of the first image frame. The determination step involves determining the three-dimensional coordinates of the target point based on the moving distance and the three-dimensional coordinates of the point pair. The first calculation step, the second calculation step, and the determination step are repeated at least once in sequence until the three-dimensional coordinates of all the target points are obtained, thus obtaining the target point cloud.

6. An image tagging device, characterized in that, include: An acquisition unit is used to acquire a target image frame, wherein the target image frame is an image frame in video data in which part of the target object is occluded; The first determining unit is configured to determine a first image frame and a second image frame based on the target image frame, wherein the first image frame is an image frame before the target image frame in which the target object is not occluded, and the second image frame is an image frame after the target image frame in which the target object is not occluded. The second determining unit is used to determine a target point cloud based on a first point cloud and a second point cloud, wherein the first point cloud is the point cloud corresponding to the target object in the first image frame, the second point cloud is the point cloud corresponding to the target object in the second image frame, and the target point cloud is the point cloud corresponding to the target object in the target image frame. A projection unit is used to project the target point cloud onto the target image frame and label it to obtain a labeled target image frame, wherein the labeled target image frame includes the labeled target object; The second determining unit includes: a matching module, used to match the points of the first point cloud and the points of the second point cloud one-to-one to obtain multiple point pairs; The determination module is used to determine the three-dimensional coordinates of multiple target points based on the three-dimensional coordinates of each point pair, thereby obtaining the target point cloud, wherein each target point corresponds one-to-one with the point pair.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program performs the method according to any one of claims 1 to 5.

8. A processor, characterized in that, The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 5 when it runs.

9. An image processing system, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising methods for performing any one of claims 1 to 5.

Images