Method and device for calculating shielding proportion of object and electronic equipment
By calculating the depth information and scale change factors of the target object in different image frames, the occlusion ratio is automatically calculated, which solves the problem that the accuracy of the occlusion degree of the labeled object depends on subjective experience, and improves the quality and usage effect of the training data.
Patent Information
- Application Number
- CN202411236660.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2025-01-10
AI Technical Summary
Under the supervised learning framework, especially in the semi-supervised and fully supervised learning situations, the accuracy of the degree of occlusion of the labeled object depends on the subjective experience of the labeled person, resulting in poor acceptance and use of the training data.
By obtaining the depth information of the target object in the initial image frame and the current image frame, the scale change factor is calculated, and the visible area of the target object is corrected based on this factor, the occlusion ratio of the target object in the current image frame is calculated, so as to automatically Accurately represent the degree of obstruction.
It improves the efficiency and accuracy of calculating the occlusion ratio of the target object, reduces the dependence on the experience of the labeler, and enhances the quality of the training data.
Smart Images
Figure CN119295529A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning, and in particular to a method, device and electronic device for calculating an object occlusion ratio. Background Art
[0002] With the rapid development of artificial intelligence and deep learning, building high-performance models increasingly relies on rich and accurate training data. In the supervised learning framework, especially in semi-supervised and fully supervised learning scenarios, the training data includes not only the original data, but also the detailed annotations of these data. The annotation information serves as a key guide for model learning, enabling the model to understand the data features and make predictions.
[0003] However, various difficulties are often encountered during the annotation process, such as judging whether the annotated object fits with other objects, the degree of occlusion of the annotated object, etc. At present, these annotation difficulties can only be judged by the annotators based on their subjective experience, which cannot guarantee the accuracy of the annotation, thus affecting the acceptance and use of the training data. Summary of the invention
[0004] In view of this, the present application provides a method, device and electronic device for calculating the occlusion ratio of an object to solve the defects existing in the related art. The technical solution of the present application is as follows:
[0005] According to an embodiment of the first aspect of the present application, a method for calculating an object occlusion ratio is provided, comprising:
[0006] Acquire an initial image frame and a current image frame containing a target object, wherein the target object is not blocked in the initial image frame;
[0007] Determining a scale change factor of the target object according to initial depth information of the target object in the initial image frame and current depth information of the target object in the current image frame, and determining a corrected visible area of the target object in the initial image frame based on the scale change factor and a current visible area of the target object in the current image frame;
[0008] The occlusion ratio of the target object in the current image frame is calculated according to the corrected visible area and the initial visible area of the target object in the initial image frame, and the occlusion ratio is used to represent the occlusion degree of the target object.
[0009] According to an embodiment of the second aspect of the present application, a device for calculating an object occlusion ratio is provided, comprising:
[0010] An acquisition unit, configured to acquire an initial image frame and a current image frame including a target object, wherein the target object is not blocked in the initial image frame;
[0011] a determining unit, configured to determine a scale change factor of the target object according to initial depth information of the target object in the initial image frame and current depth information of the target object in the current image frame, and determine a corrected visible area of the target object in the initial image frame based on the scale change factor and a current visible area of the target object in the current image frame;
[0012] A calculation unit is used to calculate the occlusion ratio of the target object in the current image frame according to the corrected visible area and the initial visible area of the target object in the initial image frame, wherein the occlusion ratio is used to characterize the degree of occlusion of the target object.
[0013] According to an embodiment of the third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.
[0014] According to an embodiment of the fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0015] According to an embodiment of the fifth aspect of the present application, a computer program product is proposed, including a computer program / instruction, and when the computer program / instruction is executed by a processor, the method described in the first aspect is implemented.
[0016] In the technical solution provided in the present application, an initial image frame and a current image frame containing a target object can be obtained. Since the target object is not blocked in the initial image frame, the initial visible area of the target object in the initial image frame can reflect the complete area of the target object. It can be understood that the position of the target object in the initial image is not the same as that in the current image frame, and the area of the target object will change in scale proportionally with the change in depth in the two image frames. Therefore, the scale change factor of the target object can be determined according to the initial depth information of the target object in the initial image frame and the current depth information in the current image frame, and the visible area of the target object in the current image frame is corrected according to the scale change factor to obtain the corrected visible area. At this time, the visible area of the target object in the current image frame is adjusted to the same measurement scale as the initial image frame, and based on the corrected visible area and the initial visible area, the occlusion ratio of the target object in the current image frame can be accurately calculated.
[0017] It can be seen that the present application can take into account the scale changes of the area of the target object in different image frames, and automatically and accurately calculate the degree of occlusion of the target object under the same measurement scale, thereby improving the efficiency and accuracy of calculating the occlusion ratio of the target object.
[0018] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and do not limit the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0020] Figure 1 is a schematic diagram of an image frame including a marked object shown in an exemplary embodiment of the present application;
[0021] Figure 2 is a flow chart of a method for calculating an object occlusion ratio shown in an exemplary embodiment of the present application;
[0022] Figure 3 is a schematic diagram of an interactive selection of an object to be identified and tracked, shown in an exemplary embodiment of the present application;
[0023] Figure 4 It is a flow chart of an overall solution shown in an exemplary embodiment of the present application;
[0024] Figure 5 is a schematic diagram of a device for calculating an object occlusion ratio shown in an exemplary embodiment of the present application;
[0025] Figure 6 It is a schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of the present application as detailed in the appended claims.
[0027] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the steps included in the method may be more or less than those described in this application. In addition, a single step described in this application may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this application may also be combined into a single step for description in other embodiments.
[0028] With the rapid development of artificial intelligence and deep learning, building high-performance models increasingly relies on rich and accurate training data. In the supervised learning framework, especially in semi-supervised and fully supervised learning scenarios, the training data includes not only the original data, but also the detailed annotations of these data. The annotation information serves as a key guide for model learning, enabling the model to understand the data features and make predictions.
[0029] Figure 1 FIG. 1 is a schematic diagram of an image frame including a labeled object shown in an exemplary embodiment of the present application. Figure 1 As shown, the image frame may include multiple annotated objects, such as annotated object 11, annotated object 12, and annotated object 13. The image frame and the annotation information of each annotated object included in the image frame can be used as training data for training the target model.
[0030] The image frame described in the embodiment of the present application can be any image frame in a video stream or a continuous frame, and the video stream or continuous frame can be either real data captured by an image acquisition device or synthetic data generated by a simulation system. The selection and definition of the annotation object is closely related to the actual scene in which the target model using the training data will be deployed, and should be an important element that can reflect the scene information. For example, in the field of autonomous driving, the annotation objects may include vehicles, pedestrians, and roads, etc.; in medical image analysis, the annotation objects may include lesions, organs, and cell tissues, etc. The following embodiments will be explained using the field of autonomous driving as an example.
[0031] Different annotated objects may have different visible states in the image frame, which can be specifically divided into unobstructed and obstructed. An annotated object may have a complete visible shape in the image frame, that is, there are no other objects or obstacles to obstruct the annotated object, and the visible shape of the annotated object is the complete shape of the annotated object at that angle, such as the annotated object 11 is in an unobstructed visible state; conversely, an annotated object may not have a complete visible shape in the image frame, that is, there are other objects or obstacles to obstruct the annotated object, and the visible shape of the annotated object is not the complete shape of the annotated object at that angle, such as the annotated object 13 obstructs part of the shape of the annotated object 12, at this time the annotated object 12 does not have a complete visible shape in the image frame, and the annotated object 12 is in an obstructed visible state.
[0032] In the process of preprocessing the training data of the target model, it may be necessary to screen the annotated objects based on the occlusion ratio of the annotated objects, such as deleting the annotation information of the annotated objects whose occlusion ratio in the image frame is greater than a preset ratio threshold, so that the target model can perform targeted learning and training based on the remaining annotated objects that meet the standard requirements and the corresponding annotation information.
[0033] However, currently, the degree of occlusion of the labeled object can only be judged by the annotator based on personal experience, which is easily affected by subjective factors. Although the relevant technology provides methods such as occlusion ratio calculation based on bounding box and occlusion ratio calculation based on template matching, the calculation accuracy cannot be guaranteed, which in turn affects the acceptance and use of training data.
[0034] In order to solve the above problem, the present application proposes a solution for calculating the occlusion ratio of an object, that is, calculating a scale change factor based on the depth information of the target object in different image frames, using the scale change factor to unify the area of the target object in different image frames to the same measurement scale, and then further calculating the occlusion ratio based on the visible area in the unobstructed state and the current visible area. The embodiments of the present application are described in detail below.
[0035] Figure 2 FIG. 1 is a flow chart of a method for calculating an object occlusion ratio according to an exemplary embodiment of the present application. Figure 2 , the method comprising:
[0036] Step 201 : acquiring an initial image frame and a current image frame including a target object, wherein the target object is not blocked in the initial image frame.
[0037] In one embodiment, the target object may be any object tracked by the SAM-Track model based on the initial segmentation result, and the initial segmentation result may be obtained in response to a user's selection operation on the initial object contained in the initial image frame.
[0038] The SAM-Track model is a framework that can accurately and effectively perform instance segmentation and tracking of any object in a video stream or continuous frames. It integrates the image segmentation model SAM (Segment Anything Model), the target tracking model DeAOT, and the text-image multimodal target detection model Grounding-DINO. It allows users to use an interactive method to select objects to be identified, and achieve high-precision tracking and instance segmentation of the selected objects in video streams or continuous frames.
[0039] Specifically, the user can find an image frame in the data set where the object to be identified and tracked is not obscured, and the SAM-Track model will use this image frame as the initial image frame. The SAM-Track model will then provide the user with some action prompts so that the user can interactively select the object to be identified and tracked in the initial image frame based on the action prompts. The user can select by clicking, selecting a box, or using text prompts. Figure 3 As shown in the figure, users can manually click or select the target to be identified, such as cars, buses, pedestrians, etc., or enter descriptive text, such as "car" or "bus", which has been classified and defined and can be understood by the system to select the target. The SAM-Track model will automatically segment the target to be identified and tracked in the initial image frame based on the user's selection results to obtain the initial segmentation result.
[0040] At the same time, since data collection is usually carried out continuously, a series of continuous images can be intercepted from the initial image frame as a continuous frame for segmentation and target tracking. After obtaining the initial segmentation result for the initial image frame and the continuous frames starting from the initial image frame, the SAM-Track model will start from the first frame after the initial image frame and perform continuous segmentation and tracking frame by frame for the selected object to be identified and tracked.
[0041] In the embodiment of the present application, the target object is any object tracked by the SAM-Track model based on the initial segmentation result, that is, any object selected by the user for identification and tracking based on the initial image frame. For any target object, the initial image frame is an image frame containing the target object and without occlusion, and the current image frame is any image frame containing the target object in a continuous frame captured previously.
[0042] It should be noted that the solution of the present application is a technical solution implemented for a single target object. When the current image frame contains multiple objects segmented and tracked by the SAM-Track model, the technical solution of the present application is applied to calculate their occlusion areas respectively.
[0043] It should also be noted that if the initial image frame and the current image frame are acquired based on an image acquisition device, then the shooting parameters when acquiring the initial image frame should be the same as the shooting parameters when acquiring the current image frame; similarly, if the initial image frame and the current image frame are generated by a simulation system, then the simulation system should also use the same rendering parameters to generate the initial image frame and the current image frame to ensure that the scale change factor calculated by the subsequent embodiments can accurately reflect the scale change ratio of the object. Here, the shooting parameters and rendering parameters are used to characterize parameters that may affect the depth information, such as focal length, position and posture of the camera model in the image acquisition device or simulation system, etc.
[0044] Step 202, determining a scale change factor of the target object according to initial depth information of the target object in the initial image frame and current depth information of the target object in the current image frame, and determining a corrected visible area of the target object in the initial image frame based on the scale change factor and the current visible area of the target object in the current image frame.
[0045] In one embodiment, each image frame has a corresponding image annotation result, which is output and stored by the 3D annotation platform. The 3D annotation platform enables users to accurately annotate the objects contained in each image frame to generate image annotation results for the object. In any image annotation result, the object annotation results of each object in the image are included. Specifically, the image annotation result of the initial image frame includes the object annotation results of each object in the initial image, and the image annotation result of the current image frame includes the object annotation results of each object in the current image. For any object annotation result, it can be identified by the object ID of the object, and the object annotation result can include the 3D annotation box of the object and the corresponding attribute information. Specifically, the 3D annotation box of the object can be described by the center point coordinates of the object (corresponding to the coordinate values of the x-axis, y-axis, and z-axis, respectively), the size of the 3D annotation box (length, width, height), and the orientation angle (also referred to as the attitude angle or rotation angle), and the attribute information can include the category information, truncation ratio, occlusion ratio, etc. of the object that can be used to characterize the attributes of the object.
[0046] It can be understood that in three-dimensional space, the x-axis, y-axis, and z-axis form a rectangular coordinate system, which are perpendicular to each other and can define the position of any point in three-dimensional space. The x-axis usually represents horizontal left and right movement, the y-axis usually represents vertical up and down movement, and the z-axis is used to represent the depth direction, that is, inward or outward movement. Generally speaking, the direction away from the observation point is the positive direction of the z-axis. The larger the coordinate corresponding to the z-axis, the farther the object is visually from the observation point, that is, the deeper the depth position of the object in the scene.
[0047] Therefore, the depth information of the target object, that is, the z-axis coordinate corresponding to the center point of the object, can be obtained from the object annotation result of the target object. Specifically, the initial depth information of the target object in the initial image frame can be obtained from the image annotation result of the initial image frame, and the current depth information of the target object in the current image frame can be obtained from the image annotation result of the current image frame.
[0048] In one embodiment, for the target object in the current image frame, the SAM-Track model can determine the object corresponding to the target object in the initial image frame through the identity information library maintained by the DeAOT model and obtain the object annotation result of the object accordingly, and obtain its depth information from the object annotation result as the initial depth information of the target object.
[0049] However, although the SAM-Track model can segment and track each object in the initial image frame based on the initial segmentation results, the object ID automatically assigned to each object is different from the object ID stored on the 3D annotation platform for identifying the object annotation results, resulting in the inability to directly match the object annotation results of the target object, and thus unable to obtain the depth information of the target object. Figure 1 The image frame shown is that the object ID automatically assigned by the SAM-Track model to the labeled object 11 may be "0001", while the object ID of the labeled object 11 on the 3D labeling platform may be "3982". Even if the SAM-Track model can determine the object corresponding to the target object in the initial image frame based on the object ID "3982" assigned to it, it cannot directly obtain the depth information of the object. In the related art, the user can manually input the object ID of each object to be identified and tracked on the 3D labeling platform based on the initial segmentation result, so that the SAM-Track model can directly obtain the object labeling result of the corresponding object according to each object ID after segmenting and tracking each object. However, this method relies entirely on manual operation, has a low degree of automation, and the efficiency and accuracy of matching cannot be guaranteed.
[0050] In one embodiment, the present application scheme can respectively calculate the distance between the center point coordinates of the target object and the center point coordinates of the object corresponding to each object annotation result, and obtain multiple distance values for the target object. Then, the object annotation result corresponding to the minimum distance can be determined as the object annotation result of the target object.
[0051] Specifically, the center point coordinates of the target object can be determined by the segmented pixels of the object obtained after segmentation and tracking based on the SAM-Track model. Specifically, the center of mass of the object can be obtained by averaging all the segmented pixels of the object. When calculating the distance between the center point coordinates of the target object and the center point coordinates of the objects corresponding to the annotation results of each object, the Euclidean distance can be used for specific calculation. If the center point coordinates of the target object are (x s ,y s ), the x-axis and y-axis coordinate values of the center point of the object corresponding to the object annotation result are (x l ,y l ), then the distance calculation formula between these two points is:
[0052]
[0053] After respectively calculating the distances between the center point coordinates of the target object and the center point coordinates of the objects corresponding to the multiple object annotation results, multiple distance values d can be obtained. 1 ,d 2 ……d n It can be understood that the smaller the distance between the center point coordinates of the target object and the center point coordinates of the objects corresponding to the multiple object annotation results is, the closer the two center points are, that is, the objects referred to by the two center points are more likely to be the same object, so the object annotation result corresponding to the minimum distance can be determined as the object annotation result of the target object.
[0054] It should be noted that the matching method described in the above embodiment is usually applied to the initial image frame, that is, the object ID is matched when the initial image frame and the initial segmentation result are determined, and in the current image frame, the SAM-Track model can also directly obtain the object annotation result of the target object in the current image frame based on the matched object ID, and then obtain the current depth information of the target object. Of course, technicians can also adjust the matching timing according to actual needs and circumstances, and this application does not limit this.
[0055] It can be seen from the above embodiments that the solution of the present application can automatically match the object annotation results of the target object, improve the degree of automation during matching, and at the same time improve the efficiency and accuracy of matching.
[0056] In one embodiment, it is understandable that the position of the target object in the initial image and in the current image frame may not be the same, and the visible area of the target object in the two image frames will change in scale in proportion to the change in depth. It is inaccurate to directly use the current visible area when the target object may be blocked and the initial visible area when the target object is not blocked to calculate the occlusion ratio. Therefore, the scale change factor of the target object can be determined based on the initial depth information and current depth information of the target object, and the visible area of the target object in the current image frame can be corrected based on the scale change factor to obtain the corrected visible area.
[0057] Specifically, in one embodiment, the initial depth information d 0 and the current depth information d t The ratio of is taken as the scale change factor, and the scale change factor s is obtained as follows:
[0058]
[0059] Afterwards, the current visible area A of the target object in the current image frame can be adjusted by the obtained scale change factor t , so that the visible area of the target object in the current image frame is adjusted to the same measurement scale as the initial image frame, and the corrected visible area A′ of the target object in the initial image frame is obtained t , which can also be understood as the mapping area of the current visible area of the target object in the initial image frame. Since the depth information is a one-dimensional measurement value and the area is a two-dimensional measurement value, the product of the square of the scale factor and the current visible area of the target object can be used as the corrected visible area A′ t :
[0060] A′ t =A t ·s 2
[0061] Specifically, in another embodiment, the initial depth information d 0 The square of d 0 2 and the current depth information d t The square of d t 2 The ratio of is taken as the scale change factor, and the scale change factor s is obtained as follows:
[0062]
[0063] Afterwards, the current visible area A of the target object in the current image frame can be adjusted by the obtained scale change factor t, and similarly adjust the visible area of the target object in the current image frame to the same measurement scale as the initial image frame, and obtain the corrected visible area A′ of the target object in the initial image frame t , which can also be understood as the mapping area of the current visible area of the target object in the initial image frame. At this time, since the scale change factor is obtained by the ratio of the square of the depth information and has been converted into a two-dimensional measurement value, the product of the scale change factor and the current visible area of the target object can be used as the corrected visible area A′ t :
[0064] A′ t =A t ·s
[0065] In one embodiment, the initial visible area of the target object can be determined based on its pixel points in the initial image frame, and the current visible area of the target object can also be determined based on its pixel points in the current image frame. Specifically, after the SAM-Track model completes the object segmentation, each object is represented as a set of one or more pixels. For any object, its pixel points can be represented by a binary mask, where 1 represents a pixel belonging to the object and 0 represents a pixel not belonging to the object. Then, the total number of pixels belonging to the object needs to be calculated, that is, the number of pixels with a value of 1 in the binary mask is counted. In another embodiment, the initial visible area and the current visible area of the target object can also be determined based on the boundary of the identified object. Specifically, the SAM-Track model generates a segmentation mask for each object, which can accurately show the outline of the corresponding object. After obtaining the boundary outline of the object, the area within the boundary is filled, and the filled pixels are counted to obtain the area of the object.
[0066] In the above embodiment, the area is in pixels, and the technicians can choose whether to convert it into actual units (such as square meters or square centimeters) according to actual conditions and needs, and this application does not impose any restrictions on this.
[0067] Step 203 , calculating the occlusion ratio of the target object in the current image frame according to the corrected visible area and the initial visible area of the target object in the initial image frame, wherein the occlusion ratio is used to represent the degree of occlusion of the target object.
[0068] In one embodiment, the degree of occlusion of the target object is characterized by an occlusion ratio, and the occlusion ratio is positively correlated with the degree of occlusion. The larger the occlusion ratio of the target object, the higher the degree of occlusion of the target object. After obtaining the corrected visible area and the initial visible area of the target object, since the current visible area of the current object is the visible area of the target object when it may be occluded, and the corrected visible area is the mapping area of the current visible area of the target object in the initial image frame, the corrected visible area A′ is obtained. t The ratio of the initial visible area to the initial visible area is A 0 , calculate the occlusion ratio R of the target object o :
[0069]
[0070] It can be seen from the above embodiments that the present application scheme can take into account the scale changes of the area of the target object in different image frames, and automatically and accurately calculate the degree of occlusion of the target object under the same measurement scale, thereby improving the efficiency and accuracy of calculating the occlusion ratio of the target object.
[0071] In one embodiment, as described above, the object annotation result of each object includes the occlusion ratio of the object, and correspondingly, the object annotation result of the target object includes the occlusion ratio of the target object. Therefore, after calculating the occlusion ratio, the occlusion ratio in the object annotation result of the target object can be updated to the occlusion ratio of the target object calculated by the method provided in the present application, so that more accurate occlusion ratio information is recorded in the object annotation result of each object, thereby improving the accuracy of the object annotation result.
[0072] In one embodiment, a preliminary judgment can also be made on the current depth information of the target object. When the current depth information is not greater than the depth threshold, the scale change factor of the target object is determined based on the initial depth information and the current depth information. The depth threshold is a threshold set by the technician according to the actual situation and needs, and is used to characterize the critical value for incorporating the object annotation result of the target object into the learning scope of the target model. It can be understood that if the depth information of the target object exceeds the depth threshold, it means that the distance between the target object and the observation point is large. At this time, the target model may not need to make a judgment on the target object, and it is not necessary to learn the object features of the target object based on the object annotation result of the target object. Therefore, in order to save computing resources, it is not necessary to calculate the occlusion ratio of the target object, and then more computing resources are concentrated on the object annotation results that may be learned by the target model, while improving the computing resource utilization and the computing efficiency of the occlusion ratio.
[0073] The following will be combined Figure 4, describe the overall process of this application scheme.
[0074] Step 401, the SAM-Track model segments and tracks the target object. The SAM-Track model will determine the initial segmentation result based on the object to be identified and tracked interactively selected by the user, and based on the initial segmentation result, segment and track the determined object to be identified and tracked in the captured continuous frames. Among them, any identified and tracked object can be used as the target object in the following application scheme.
[0075] Step 402, matching the object annotation information of the target object. For the initial image frame or the current image frame, respectively calculate the distance between the center point coordinates of the target object obtained by the SAM-Track model segmentation and the center point coordinates of the object corresponding to each object annotation result stored on the 3D annotation platform, and determine the object annotation result corresponding to the minimum distance as the object annotation result of the target object.
[0076] Step 403, determining the corrected visible area of the target object. Specifically, first perform step 4031, calculate the scale change factor according to the current depth information and the initial depth information of the target object; then perform step 4032, adjust the current visible area of the target object in the current image frame based on the scale change factor, so that the visible area of the target object in the current image frame is adjusted to the same measurement scale as the initial image frame, and obtain the corrected visible area of the target object in the initial image frame.
[0077] Step 404: Calculate the occlusion ratio of the target object based on the corrected visible area and the initial visible area of the target object.
[0078] Step 405: Update the calculated occlusion ratio into the object annotation result of the target object.
[0079] Step 406, determine whether all the image frames in the continuous frames have been processed through the above steps. If all the image frames have been processed, end the process; if there are still image frames that have not been processed, return to step 401 and perform the above steps again for the unprocessed image frames.
[0080] Based on the object occlusion ratio calculation method provided in this application, this application also provides an object occlusion ratio calculation device, see Figure 5 ,include:
[0081] An acquisition unit 501 is used to acquire an initial image frame and a current image frame containing a target object, wherein the target object is not blocked in the initial image frame;
[0082] a determining unit 502, configured to determine a scale change factor of the target object according to initial depth information of the target object in the initial image frame and current depth information of the target object in the current image frame, and determine a corrected visible area of the target object in the initial image frame based on the scale change factor and a current visible area of the target object in the current image frame;
[0083] The calculation unit 503 is used to calculate the occlusion ratio of the target object in the current image frame according to the corrected visible area and the initial visible area of the target object in the initial image frame, wherein the occlusion ratio is used to represent the degree of occlusion of the target object.
[0084] Optionally, the determining unit 502 is specifically configured to:
[0085] The ratio of the initial depth information to the current depth information is used as the scale change factor.
[0086] Optionally, the determining unit 502 is specifically configured to:
[0087] The product of the current visible area of the target object in the current image frame and the square of the scale change factor is determined, and the product is used as the corrected visible area of the target object in the initial image frame.
[0088] Optionally, the determining unit 502 is specifically configured to:
[0089] When the current depth information of the target object in the current image frame is not greater than a depth threshold, a scale change factor of the target object is determined according to the initial depth information of the target object in the initial image frame and the current depth information in the current image frame.
[0090] Optionally, the device further comprises:
[0091] The area determination unit 504 is used to determine the initial visible area of the target object based on the pixel points of the target object in the initial image frame; and / or determine the current visible area of the target object based on the pixel points of the target object in the current image frame.
[0092] Optionally, the target object is any object tracked by the SAM-Track model based on an initial segmentation result, and the initial segmentation result is obtained in response to a user's initial object selection operation contained in the initial image frame.
[0093] Optionally, the image annotation result of the current image frame includes object annotation results of each object in the image, wherein the object annotation result of the target object includes an occlusion ratio of the target object, and the device further includes:
[0094] The updating unit 505 is configured to update the occlusion ratio in the object annotation result of the target object to the calculated occlusion ratio.
[0095] Optionally, the object labeling result of the target object is determined by:
[0096] The distances between the center point coordinates of the target object and the center point coordinates of the objects corresponding to the respective object annotation results are calculated respectively, and the object annotation result corresponding to the minimum distance is determined as the object annotation result of the target object.
[0097] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0098] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.
[0099] Correspondingly, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any of the above embodiments when executing the computer program.
[0100] See also Figure 6 At the hardware level, the electronic device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and may also include hardware required for other services. The processor 602 reads the corresponding computer program from the non-volatile memory 610 into the memory 608 and then runs it, forming a vehicle speed adjustment device at the logical level. Of course, in addition to the software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0101] Accordingly, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any of the above embodiments is implemented.
[0102] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.
[0103] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0104] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0105] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0106] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.
[0107] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0108] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0109] The above description is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.
Claims
1. A method for calculating an object occlusion ratio, characterized in that: include: Acquire an initial image frame and a current image frame containing a target object, wherein the target object is not blocked in the initial image frame; Determining a scale change factor of the target object according to initial depth information of the target object in the initial image frame and current depth information of the target object in the current image frame, and determining a corrected visible area of the target object in the initial image frame based on the scale change factor and a current visible area of the target object in the current image frame; The occlusion ratio of the target object in the current image frame is calculated according to the corrected visible area and the initial visible area of the target object in the initial image frame, and the occlusion ratio is used to represent the occlusion degree of the target object.
2. The method according to claim 1, characterized in that The determining the scale change factor of the target object according to the initial depth information of the target object in the initial image frame and the current depth information of the target object in the current image frame comprises: The ratio of the initial depth information to the current depth information is used as the scale change factor.
3. The method according to claim 1, characterized in that The determining, based on the scale change factor and the current visible area of the target object in the current image frame, a revised visible area of the target object in the initial image frame comprises: The product of the current visible area of the target object in the current image frame and the square of the scale change factor is determined, and the product is used as the corrected visible area of the target object in the initial image frame.
4. The method according to claim 1, characterized in that: Determining a scale change factor of the target object according to initial depth information of the target object in the initial image frame and current depth information of the target object in the current image frame includes: When the current depth information of the target object in the current image frame is not greater than a depth threshold, a scale change factor of the target object is determined according to the initial depth information of the target object in the initial image frame and the current depth information in the current image frame.
5. The method according to claim 1, characterized in that Also includes: Determining an initial visible area of the target object based on pixel points of the target object in the initial image frame; and / or, A current visible area of the target object is determined based on the pixel points of the target object in the current image frame.
6. The method according to claim 1, characterized in that The target object is any object tracked by the SAM-Track model based on the initial segmentation result, and the initial segmentation result is obtained in response to the user's initial object selection operation contained in the initial image frame.
7. The method according to claim 1, characterized in that The image annotation result of the current image frame includes object annotation results of each object in the image, wherein the object annotation result of the target object includes an occlusion ratio of the target object, and the method further includes: The occlusion ratio in the object annotation result of the target object is updated to the calculated occlusion ratio.
8. The method according to claim 7, characterized in that The object labeling result of the target object is determined by: The distances between the center point coordinates of the target object and the center point coordinates of the objects corresponding to the respective object annotation results are calculated respectively, and the object annotation result corresponding to the minimum distance is determined as the object annotation result of the target object.
9. A device for calculating an object occlusion ratio, characterized in that: include: An acquisition unit, configured to acquire an initial image frame and a current image frame including a target object, wherein the target object is not blocked in the initial image frame; a determining unit, configured to determine a scale change factor of the target object according to initial depth information of the target object in the initial image frame and current depth information of the target object in the current image frame, and determine a corrected visible area of the target object in the initial image frame based on the scale change factor and a current visible area of the target object in the current image frame; A calculation unit is used to calculate the occlusion ratio of the target object in the current image frame according to the corrected visible area and the initial visible area of the target object in the initial image frame, wherein the occlusion ratio is used to characterize the degree of occlusion of the target object.
10. An electronic device, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 8 by executing the executable instructions.
11. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
12. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.