Continuous frame image feature determination method, labeling method, system, device and medium

Abstract

The application provides a continuous frame image feature determination method, a labeling method, a system, a device and a medium. The continuous frame image feature determination method comprises the following steps: acquiring continuous frame image data of a feature to be determined, time points corresponding to each frame of image data in the continuous frame image data, and a real-time relative vehicle speed of a vehicle speed of a vehicle to be controlled corresponding to the continuous frame image data of the feature to be determined relative to a target object; determining first frame image data according to the time points; selecting the first frame image data as reference data; mapping the reference data to a coordinate system in which the continuous frame image data is located for calibration to obtain reference coordinates of the reference data at the corresponding time points; and determining continuous frame image features according to the reference coordinates, the real-time relative vehicle speed and the time points corresponding to each frame of image data in the continuous frame image data. According to the application, only the first frame positioning labeling is required, and the continuous frame labeling can be realized after the position calibration, thereby improving the labeling efficiency.

Description

Technical Field

[0001] This application relates to the field of intelligent control technology, specifically to methods, annotation methods, systems, devices, and media for determining features of continuous frame images. Background Technology

[0002] Autonomous driving algorithms are a crucial component of autonomous driving technology. Developing these algorithms requires multiple incremental training iterations to improve vehicle-side performance. Constructing the vehicle-side algorithm model necessitates a large amount of well-labeled 2D images and 3D radar point cloud image data as datasets (including training and validation datasets). The quality of the dataset directly impacts the accuracy of the trained algorithm. Therefore, improving the labeling quality of point cloud image data is of great significance for enhancing the accuracy of the resulting perception algorithm model.

[0003] In related technologies, the annotation of continuous frame images is accomplished either through manual full-element annotation or through model pre-labeling. Specifically, a large model is trained using labeled bounding boxes or real-world image segmentation instances as the training set. The trained model is then validated using a validation dataset to check its parameters; this model serves as the first generation. Pre-labeling results are then generated, and the resulting files are sent to the manual annotation stage for correction and completion. In the actual pre-labeling process, images with unsatisfactory annotation results are selected for manual annotation and used as a new dataset to iterate the model for the next generation, continuously iterating to improve the prediction accuracy of the large model. However, this method of continuous frame image annotation is costly, and its efficiency cannot meet the algorithm's training requirements. Developing large models using this annotation method would inevitably require a significant investment of R&D manpower, have a long model iteration cycle, and the pre-labeling results may not fundamentally reduce the labor intensity of manual annotation. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the present invention provides a method, annotation method, system, device and medium for determining features of continuous frame images, so as to solve the technical problem of low annotation efficiency of continuous frame images.

[0005] This application provides a method for determining features of consecutive frame images, the method comprising:

[0006] Acquire continuous frame image data of the feature to be determined, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data of the feature to be determined.

[0007] The first frame image data is determined based on the time point, and the first frame image data is selected as the reference data.

[0008] The reference data is mapped to the coordinate system where the continuous frame image data is located for calibration, so as to obtain the reference coordinates of the reference data at the corresponding time point;

[0009] The continuous frame image features are determined based on the reference coordinates, the real-time relative vehicle speed, and the time points corresponding to each frame of the continuous frame image data.

[0010] In an exemplary embodiment of this application, determining the features of consecutive frames of images includes:

[0011] Based on the reference coordinates, the real-time relative vehicle speed, and the time points corresponding to each frame of image data in the continuous frame image data, the predicted coordinates of each point in each continuous frame image data, excluding the first frame image data, are determined in chronological order.

[0012] By connecting the predicted coordinates of each point in the same frame of data, we can obtain the features of the next adjacent frame of data in each consecutive frame of image data.

[0013] In an exemplary embodiment of this application, determining the predicted coordinates of each point in each consecutive frame of image data, excluding the first frame image data, includes:

[0014] Determine the offset angle based on the reference coordinates;

[0015] Based on the offset angle, real-time relative vehicle speed, reference coordinates, preset frame rate, and the time point corresponding to each frame of the continuous frame image data, the predicted coordinates of each point in each continuous frame image data are determined.

[0016] In one exemplary embodiment of this application, determining the offset angle includes:

[0017] Determine the offset parameters based on the reference coordinates;

[0018] The offset angle is determined based on the offset parameters and reference coordinates.

[0019] The time interval between consecutive frames is determined based on the preset frame rate;

[0020] Based on the offset angle, real-time relative vehicle speed, reference coordinates, time interval, and the time point corresponding to each frame of image data in the continuous frame image data, the predicted coordinates of each point in each continuous frame image data are determined.

[0021] In an exemplary embodiment of this application, determining the predicted coordinates of each point in each consecutive frame of image data includes:

[0022] Based on the offset angle, real-time relative vehicle speed, reference coordinates, time interval, and the time point corresponding to each frame of image data in the continuous frame image data, determine the horizontal and vertical displacements of the next adjacent frame of image data of the first frame image data.

[0023] By combining the horizontal and vertical displacement offsets, the predicted coordinates of each point in the second frame image data are obtained.

[0024] The corresponding offset angle is determined sequentially based on the predicted coordinates and offset parameters of each point in the adjacent previous frame image data;

[0025] The predicted coordinates of each point in the adjacent next frame of image data are determined sequentially based on the corresponding offset angle, real-time relative vehicle speed, predicted coordinates of each point in the adjacent previous frame of image data, preset frame rate, and time points corresponding to each frame of image data in the continuous frame of image data.

[0026] Secondly, this application provides a method for annotating consecutive frame images, the method comprising:

[0027] Acquire the continuous frame image data to be labeled, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data to be labeled.

[0028] The first frame image data is determined based on the time point, and the first frame image data is selected as the reference data.

[0029] The reference data is mapped to the coordinate system where the continuous frame image data is located for calibration, so as to obtain the reference coordinates of the reference data at the corresponding time point;

[0030] The continuous frame image features are determined based on the reference coordinates, the real-time relative vehicle speed, and the time points corresponding to each frame of image data in the continuous frame image data.

[0031] The features of the consecutive frames of images are classified and labeled to complete the labeling.

[0032] Thirdly, this application provides a continuous frame image feature determination system, the continuous frame image feature determination system comprising:

[0033] The acquisition module is used to acquire continuous frame image data of the feature to be determined, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data of the feature to be determined.

[0034] The reference data selection module is used to determine the first frame image data based on the time point and select the first frame image data as reference data.

[0035] The first determining module is used to map the reference data to the coordinate system where the continuous frame image data is located for calibration, so as to obtain the reference coordinates of the reference data at the corresponding time point;

[0036] The second determining module is used to determine the continuous frame image features based on the reference coordinates, the real-time relative vehicle speed, and the time points corresponding to each frame image data in the continuous frame image data.

[0037] Fourthly, this application provides a continuous frame image annotation system, the continuous frame image annotation system comprising:

[0038] The acquisition module is used to acquire continuous frame image data to be labeled, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data to be labeled.

[0039] The reference data selection module is used to determine the first frame image data based on the time point and select the first frame image data as reference data.

[0040] The first determining module is used to map the reference data to the coordinate system where the continuous frame image data is located for calibration, so as to obtain the reference coordinates of the reference data at the corresponding time point;

[0041] The second determining module is used to determine the continuous frame image features based on the reference coordinates, the real-time relative vehicle speed, and the time points corresponding to each frame image data in the continuous frame image data.

[0042] The annotation module is used to perform category recognition and annotation on the features of the continuous frame images, and complete the annotation.

[0043] In another aspect, this application provides an electronic device, the electronic device comprising:

[0044] One or more processors;

[0045] A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the method described above.

[0046] In another aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the method described above.

[0047] The beneficial effects of this invention are:

[0048] Compared to traditional pre-labeling methods, this application eliminates the reliance on large models for precise targeted training, thereby achieving pre-labeling and reducing the complexity of the labeling algorithm.

[0049] Compared to traditional pre-annotation methods that require manual intervention for completion and modification, this application, based on the differential method, can accurately calculate the offset of coordinate changes within a small displacement range, thereby achieving accurate coordinate point prediction.

[0050] This application only requires positioning and annotation of the first frame to achieve position calibration, and then continuous frame annotation can be achieved, thus improving annotation efficiency.

[0051] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0053] Figure 1 A flowchart illustrating a method for determining features of consecutive frames of images, as shown in an exemplary embodiment of this application;

[0054] Figure 2 for Figure 1 The flowchart of determining the features of consecutive frames in step S140 in the illustrated embodiment is shown in an exemplary embodiment.

[0055] Figure 3 for Figure 2 The flowchart in the illustrated embodiment shows the process of determining the predicted coordinates of each point in each consecutive frame image data except for the first frame image data in step S210, as described in an exemplary embodiment.

[0056] Figure 4 for Figure 3 The flowchart of determining the offset angle in step S130 in the illustrated embodiment is shown in an exemplary embodiment.

[0057] Figure 5 for Figure 4 The schematic diagram of the continuous frame linear offset compensation principle in the embodiment shown is as follows;

[0058] Figure 6 for Figure 3 The flowchart in the illustrated embodiment shows the process of determining the predicted coordinates of each point in each consecutive frame of image data in step S320, as described in an exemplary embodiment.

[0059] Figure 7 for Figure 6 The flowchart in the illustrated embodiment shows the process of determining the predicted coordinates of each point in each consecutive frame of image data in step S620, as described in an exemplary embodiment.

[0060] Figure 8 A flowchart illustrating a continuous frame image annotation method as an exemplary embodiment of this application;

[0061] Figure 9 A flowchart illustrating a continuous frame image annotation method according to a specific embodiment;

[0062] Figure 10 A block diagram illustrating a continuous frame image feature determination system as an exemplary embodiment of this application;

[0063] Figure 11 A block diagram illustrating a continuous frame image annotation system as an exemplary embodiment of this application;

[0064] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0065] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0066] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0067] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0068] Please see Figure 1 , Figure 1A flowchart illustrating a method for determining features of consecutive frames of an image, as shown in an exemplary embodiment of this application.

[0069] like Figure 1 As shown in an exemplary embodiment of this application, the continuous frame image feature determination method includes at least steps S110, S120, S130, and S140, which are described in detail below:

[0070] Step S110. Obtain continuous frame image data of the feature to be determined, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data of the feature to be determined.

[0071] Specifically, real-time relative vehicle speed can be obtained through equipment such as radar.

[0072] Step S120. Determine the first frame image data based on the time point, and select the first frame image data as reference data;

[0073] Specifically, the time sequence of each frame of image data can be determined based on the time point, and the earliest data, i.e. the first frame of image data, can be determined based on the time sequence.

[0074] Step S130. Map the reference data to the coordinate system where the continuous frame image data is located for calibration, and obtain the reference coordinates of the reference data at the corresponding time point;

[0075] Step S140. Determine the features of the continuous frame images based on the reference coordinates, real-time relative vehicle speed, and the time points corresponding to each frame of image data in the continuous frame image data.

[0076] In related technologies, the annotation of continuous frame images is completed either through manual full-element annotation or through model pre-brushing. After analyzing the above solutions, the inventors found that using this method for continuous frame image annotation is costly and the annotation efficiency is insufficient to meet the training requirements of the algorithm. If this annotation method is used for the development of large models, it will inevitably require a large investment of R&D manpower, the model iteration cycle will be long, and the pre-annotation results may not fundamentally reduce the labor intensity of manual annotation. Therefore, considering the acquisition of continuous frame image data to be labeled, the time points corresponding to each frame in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object, the first frame image data is determined based on the time points. The first frame image data is selected as reference data and mapped to the coordinate system of the continuous frame image data for calibration, obtaining the reference coordinates of the reference data at the corresponding time points. Based on the reference coordinates, real-time relative speed, and the time points corresponding to each frame in the continuous frame image data, the continuous frame image features are determined, and annotation is performed based on the determined continuous frame image data. Compared with traditional pre-annotation methods, this application eliminates the dependence on large models with precise directional training, thereby achieving pre-annotation and reducing the complexity of the annotation algorithm. Compared with the manual intervention and modification operations in traditional pre-annotation, this application, based on the differential method, can achieve accurate offset calculation of coordinate changes within a small displacement range, thereby achieving accurate coordinate point prediction. This application only needs to locate and annotate the first frame to achieve position calibration, and then the continuous frame annotation can be achieved, thereby improving the annotation efficiency.

[0077] Please see Figure 2 , Figure 2 for Figure 1 The flowchart of determining the features of consecutive frames in step S140 in the illustrated embodiment is shown in an exemplary embodiment.

[0078] like Figure 2 As shown in an exemplary embodiment of this application, Figure 1 In the illustrated embodiment, the process of determining the features of consecutive frame images in step S140 includes steps S210 and S220, which are described in detail below:

[0079] Step S210. Based on the reference coordinates, real-time relative vehicle speed, and the time points corresponding to each frame of image data in the continuous frame image data, determine the predicted coordinates of each point in each continuous frame image data except for the first frame image data in chronological order.

[0080] Step S220. Connect the predicted coordinates of each point in the same frame of data to obtain the features of the next adjacent frame of data in each consecutive frame of image data.

[0081] Specifically, since the time interval between adjacent frames is extremely short, the features of the next adjacent frame can be determined by the predicted coordinates of the previous frame. That is, the predicted coordinates of each point in the same frame are connected to form a graph, and the features of the next adjacent frame in each consecutive frame of image data are obtained.

[0082] Please see Figure 3 , Figure 3 for Figure 2 The flowchart in the illustrated embodiment shows the process of determining the predicted coordinates of each point in each consecutive frame image data except for the first frame image data in step S210, in an exemplary embodiment.

[0083] like Figure 3 As shown in an exemplary embodiment of this application, Figure 2 In the illustrated embodiment, step S210, which determines the predicted coordinates of each point in each consecutive frame of image data (excluding the first frame), includes steps S310 and S320, which are described in detail below:

[0084] Step S310. Determine the offset angle based on the reference coordinates;

[0085] Step S320. Determine the predicted coordinates of each point in each continuous frame image data based on the offset angle, real-time relative vehicle speed, reference coordinates, preset frame rate, and the time point corresponding to each frame image data in the continuous frame image data.

[0086] Please see Figure 4 , Figure 4 for Figure 3 The flowchart of determining the offset angle in step S310 of the illustrated embodiment is shown in an exemplary embodiment.

[0087] like Figure 4 As shown in an exemplary embodiment of this application, Figure 3 In the illustrated embodiment, the process of determining the offset angle in step S310 includes steps S410 and S420, which are described in detail below:

[0088] Step S410. Determine the offset parameters based on the reference coordinates;

[0089] Specifically, such as Figure 5 As shown, the position of the target object does not change much between two adjacent frames. The differential method is used to take (t, t+1) time as a differential unit. The direction of position change of the same particle is the direction of relative velocity ΔV. Considering that a small vertical displacement will cause errors in the position offset, the angle θ between the direction of relative velocity and the horizontal direction is used to achieve linear compensation by decomposing the horizontal and vertical directions. The horizontal displacement is ΔV·ΔTcosθ, and the vertical displacement is ΔV·ΔTsinθ.

[0090] The coordinates of the first frame, i.e., the reference coordinates (the reference coordinates are in a rectangular coordinate system), are determined. Therefore, ΔX = ΔV·ΔTcosθ = 1 / f·ΔV cosθ; ΔY = ΔV·ΔT sinθ = 1 / f·ΔV sinθ. Then x... 1t+1 =x 1t +ΔX, y 1t+1 =y 1t +ΔY, within the differential interval, we can obtain the curve f(x,θ) with respect to θ. Since the vehicle speed changes continuously and the offset between adjacent frames is small, the curve f(x,θ) can be represented by a polynomial expansion or series expansion. Where θ is the constant term a n The parameters.

[0091] In a Cartesian coordinate system, the offset points of any particle in consecutive frames at different times should satisfy the equation of the curve f(x,θ). Then the point set q={(x 1t ,y 1t ),(x 1t+1 ,y 1t+1 ),(x 1t+2 ,y 1t+2 ),…,(x 1t+k ,y 1t+k )} satisfies f(x,θ), according to the recurrence relation

[0092]

[0093]

[0094] Because the coordinates of the first frame (x 1t ,y 1t Since it has been determined, the curve equation containing parameter θ can be calculated based on each point in the point set q, and then the offset parameters in the polynomial can be solved.

[0095] Step S420. Determine the offset angle based on the offset parameters and reference coordinates.

[0096] Specifically, such as Figure 5 As shown, based on the calculated fitted curve equation, the θ parameter needs to be solved. Since the displacement curve is differentiable for two adjacent frames (t, t+1), the ratio of the vertical displacement to the horizontal displacement is exactly the slope of the displacement curve, i.e., tanθ = ΔY / ΔX. Therefore:

[0097]

[0098]

[0099] The θ value, i.e. the offset angle, of each predicted coordinate point can be calculated using formula (IV).

[0100] Please see Figure 6 , Figure 6 for Figure 3 The flowchart in an exemplary embodiment shows the process of determining the predicted coordinates of each point in each consecutive frame image data in step S320.

[0101] like Figure 6 As shown in an exemplary embodiment of this application, Figure 3 In the illustrated embodiment, step S320, which involves determining the predicted coordinates of each point in the consecutive frame image data, includes steps S610 and S620, which are detailed below:

[0102] Step S610. Determine the time interval between consecutive frames according to the preset frame rate;

[0103] Specifically, the time interval between consecutive frames is determined according to formula (V) based on the preset frame rate:

[0104] ΔT = 1 / f(V);

[0105] Where ΔT is the time interval in seconds (s) and f is the preset frame rate in frames per second (fps).

[0106] Step S620. Determine the predicted coordinates of each point in each consecutive frame image data based on the offset angle, real-time relative vehicle speed, reference coordinates, time interval, and the time point corresponding to each frame image data in the consecutive frame image data.

[0107] Please see Figure 7 , Figure 7 for Figure 6 The flowchart in an exemplary embodiment shows the process of determining the predicted coordinates of each point in each consecutive frame image data in step S620.

[0108] like Figure 7 As shown in an exemplary embodiment of this application, Figure 6 In the illustrated embodiment, the process of determining the predicted coordinates of each point in each consecutive frame of image data in step S620 includes steps S710, S720, S730, and S740, which are described in detail below:

[0109] Step S710. Determine the horizontal and vertical displacements of the second frame image data based on the offset angle, real-time relative vehicle speed, reference coordinates, time interval, and the time points corresponding to each frame image data in the continuous frame image data.

[0110] Specifically, the horizontal and vertical displacements of the second frame image data can be calculated according to formulas (I) and (II) above.

[0111] Step S720. Combine the horizontal displacement offset and the vertical displacement offset to obtain the predicted coordinates of each point in the second frame image data;

[0112] Step S730. Determine the corresponding offset angle according to the predicted coordinates and offset parameters of each point in the adjacent previous frame image data;

[0113] Step S740. Based on the corresponding offset angle, real-time relative vehicle speed, predicted coordinates of each point in the adjacent previous frame image data, preset frame rate, and time points corresponding to each frame image data in the continuous frame image data, determine the predicted coordinates of each point in the adjacent next frame image data.

[0114] Please see Figure 8 , Figure 8 A flowchart illustrating a continuous frame image annotation method as an exemplary embodiment of this application.

[0115] like Figure 8 As shown in an exemplary embodiment of this application, the continuous frame image annotation method includes at least steps S810, S820, S830, S840, and S850, which are described in detail below:

[0116] Step S810. Obtain the continuous frame image data to be labeled, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data to be labeled.

[0117] Step S820. Determine the first frame image data based on the time point, and select the first frame image data as reference data;

[0118] Step S830. Map the reference data to the coordinate system where the continuous frame image data is located for calibration, and obtain the reference coordinates of the reference data at the corresponding time point;

[0119] Step S840. Determine the continuous frame image features based on the reference coordinates, real-time relative vehicle speed, and the time points corresponding to each frame image data in the continuous frame image data;

[0120] Step S850. Perform category identification and annotation on the features of consecutive frame images to complete the annotation.

[0121] Please see Figure 9 In one specific embodiment, the steps of the continuous frame image annotation method are as follows:

[0122] Step S910. Obtain the continuous frame image data to be labeled, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed ΔV of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data to be labeled, wherein the preset frame rate is f.

[0123] Establish a Cartesian coordinate system based on the first frame image data or point cloud. In this system, time t, t+1, and t+2 represent the time points of the first, second, and third frames, respectively. The coordinates of the first frame image data at time t are (x...). 1t ,y 1t The coordinates corresponding to the second frame image data at time t+1 are (x... 1t+1 ,y 1t+1 The coordinates of the third frame of data at time t+2 are (x... 1t+2 ,y 1t+2 );

[0124] According to formula (I), the time interval between consecutive frames is determined based on the preset frame rate:

[0125] ΔT = 1 / f(I);

[0126] Where ΔT is the time interval in seconds (s), and f is the preset frame rate in frames per second (fps);

[0127] Step S920. Label the first frame of image data. The label set Set corresponds to the set of coordinate points at times t, t+1, ..., t+k. The set of coordinate points for each frame in a series of frames is described as follows:

[0128] Set t =((x) 1t ,y 1t ),(x 2t ,y 2t ),(x 3t ,y 3t )…(x nt ,y nt ))(T=t) (VI);

[0129] Set t+1 =((x) 1t+1 ,y 1t+1 ),(x 2t+1 ,y 2t+1 ),(x 3t+1 ,y 3t+1 )…(x nt+1 ,y nt+1 (VII); ))(T=t+1)

[0130] ...

[0131] Set t+k =((x) 1t+k ,y 1t+k ),(x 2t+k ,y 2t+k ),(x3t+k ,y 3t+k )…(x nt+k ,y nt+k ))(T=t+k)(VIII);

[0132] The reference data is mapped onto the established Cartesian coordinate system for calibration, yielding the reference coordinates (x, y) of the reference data at time t. 1t ,y 1t That is, the coordinates of the first frame;

[0133] Step S930: As Figure 5 As shown, the position of the target object does not change much between two adjacent frames. The differential method is used to take (t, t+1) time as a differential unit. The direction of position change of the same particle is the direction of relative velocity ΔV. Considering that a small vertical displacement will cause errors in the position offset, the angle θ between the direction of relative velocity and the horizontal direction is used to achieve linear compensation by decomposing the horizontal and vertical directions. The horizontal displacement is ΔV·ΔTcosθ, and the vertical displacement is ΔV·ΔTsinθ.

[0134] Step S940: The coordinates of the first frame have been determined, so ΔX = ΔV·ΔT cosθ = 1 / f·ΔV cosθ; ΔY = ΔV·ΔTsinθ = 1 / f·ΔV sinθ, then x 1t+1 =x 1t +ΔX, y 1t+1 =y 1t +ΔY, within the differential interval, we can obtain the curve f(x,θ) with respect to θ. Since the vehicle speed changes continuously and the offset between adjacent frames is small, the curve f(x,θ) can be represented by a polynomial expansion or series expansion. Where θ is the constant term a n The parameters.

[0135] In a Cartesian coordinate system, the offset points of any particle in consecutive frames at different times should satisfy the equation of the curve f(x,θ). Then the point set q={(x 1t ,y 1t ),(x 1t+1 ,y 1t+1 ),(x 1t+2 ,y 1t+2 ),…,(x 1t+k ,y 1t+k )} satisfies f(x,θ), according to the recurrence relation

[0136]

[0137]

[0138] Because the coordinates of the first frame (x 1t ,y1t The equation of the curve containing the parameter θ can be calculated based on each point in the point set q.

[0139] Based on the calculated fitted curve equation, it is necessary to solve for the θ parameter. Since the displacement curve is differentiable between two adjacent frames (t, t+1), it can be known that the ratio of the vertical displacement offset to the horizontal displacement offset is exactly the slope of the displacement curve, i.e., tanθ = ΔY / ΔX.

[0140]

[0141]

[0142] The θ value of each predicted coordinate point can be calculated according to formula (IV). Substituting the calculated θ value into formulas (I) and (II) will yield the coordinate point of the next adjacent frame, i.e., the predicted coordinate.

[0143] Step S950: Connect the predicted coordinates to form a graph to obtain the continuous frame image features of the next adjacent frame. Match the label to identify and label the continuous frame image features, and then output the labeling results of the continuous frames in JSON format and store them on disk.

[0144] Please see Figure 10 This application also provides a continuous frame image feature determination system M1000, which includes:

[0145] The acquisition module M1010 is used to acquire continuous frame image data of the feature to be determined, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data of the feature to be determined.

[0146] The reference data selection module M1020 is used to determine the first frame image data based on the time point and select the first frame image data as reference data;

[0147] The first determining module M1030 is used to map the reference data to the coordinate system where the continuous frame image data is located for calibration, so as to obtain the reference coordinates of the reference data at the corresponding time point.

[0148] The second determining module M1040 is used to determine the features of continuous frame images based on the reference coordinates, real-time relative vehicle speed, and the time points corresponding to each frame of image data in the continuous frame image data.

[0149] Please see Figure 11 This application also provides a continuous frame image annotation system M1100, which includes:

[0150] The acquisition module M1110 is used to acquire continuous frame image data to be labeled, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data to be labeled.

[0151] The reference data selection module M1120 is used to determine the first frame image data based on the time point and select the first frame image data as reference data.

[0152] The first determining module M1130 is used to map the reference data to the coordinate system where the continuous frame image data is located for calibration, so as to obtain the reference coordinates of the reference data at the corresponding time point.

[0153] The second determining module M1140 is used to determine the features of the continuous frame image based on the reference coordinates, real-time relative vehicle speed and the time points corresponding to each frame image data in the continuous frame image data.

[0154] The annotation module M1150 is used to perform category recognition and annotation of features in consecutive frames of images, thus completing the annotation process.

[0155] It should be noted that the continuous frame image feature determination system and the continuous frame image feature determination method provided in the above embodiments belong to the same concept, as do the continuous frame image annotation system and the continuous frame image annotation method provided in the above embodiments. The specific methods by which each module and unit performs its operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the continuous frame image feature determination system and the continuous frame image annotation system provided in the above embodiments can be configured to have different functional modules assigned to them as needed. That is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0156] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the continuous frame image feature determination method or continuous frame image annotation method provided in the above embodiments.

[0157] Figure 12 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 12 The computer system 1200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0158] like Figure 12As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203, such as performing the methods described in the above embodiments. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0159] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0160] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0161] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0163] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0164] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the continuous frame image feature determination method or the continuous frame image annotation method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0165] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the continuous frame image feature determination method or continuous frame image annotation method provided in the various embodiments described above.

[0166] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for determining features of consecutive frame images, characterized in that, The method for determining the features of consecutive frames of images includes: Acquire continuous frame image data of the feature to be determined, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data of the feature to be determined. The temporal order of each frame of image data is determined based on the time point. The earliest data is determined based on the temporal order. The earliest data is the first frame of image data, which is selected as the reference data. The reference data is mapped to the coordinate system where the continuous frame image data is located for calibration, so as to obtain the reference coordinates of the reference data at the corresponding time point; Determine the offset angle based on the reference coordinates; The time interval between consecutive frames is determined based on the preset frame rate; Based on the offset angle, real-time relative vehicle speed, reference coordinates, time interval, and the time points corresponding to each frame of image data in the continuous frame image data, determine the horizontal and vertical displacement offsets of the second frame image data. By combining the horizontal and vertical displacement offsets, the predicted coordinates of each point in the second frame image data are obtained. The corresponding offset angle is determined sequentially based on the predicted coordinates and offset parameters of each point in the adjacent previous frame image data; The predicted coordinates of each point in the adjacent next frame of image data are determined sequentially based on the corresponding offset angle, real-time relative vehicle speed, predicted coordinates of each point in the adjacent previous frame of image data, preset frame rate, and time points corresponding to each frame of image data in the continuous frame of image data. By connecting the predicted coordinates of each point in the same frame of data, we can obtain the features of the next adjacent frame of data in each consecutive frame of image data.

2. The method for determining features of consecutive frames of images according to claim 1, characterized in that, Determine the offset angle, including: Determine the offset parameters based on the reference coordinates; The offset angle is determined based on the offset parameters and reference coordinates.

3. A method for annotating consecutive frame images, characterized in that, The continuous frame image annotation method includes: Acquire the continuous frame image data to be labeled, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data to be labeled. The temporal order of each frame of image data is determined based on the time point. The earliest data is determined based on the temporal order. The earliest data is the first frame of image data, which is selected as the reference data. The reference data is mapped to the coordinate system where the continuous frame image data is located for calibration, so as to obtain the reference coordinates of the reference data at the corresponding time point; Determine the offset angle based on the reference coordinates; The time interval between consecutive frames is determined based on the preset frame rate; Based on the offset angle, real-time relative vehicle speed, reference coordinates, time interval, and the time points corresponding to each frame of image data in the continuous frame image data, determine the horizontal and vertical displacement offsets of the second frame image data. By combining the horizontal and vertical displacement offsets, the predicted coordinates of each point in the second frame image data are obtained. The corresponding offset angle is determined sequentially based on the predicted coordinates and offset parameters of each point in the adjacent previous frame image data; The predicted coordinates of each point in the adjacent next frame of image data are determined sequentially based on the corresponding offset angle, real-time relative vehicle speed, predicted coordinates of each point in the adjacent previous frame of image data, preset frame rate, and time points corresponding to each frame of image data in the continuous frame of image data. By connecting the predicted coordinates of each point in the same frame of data, the features of the next adjacent frame of data in each consecutive frame of image data are obtained. The features of the consecutive frames of images are classified and labeled to complete the labeling.

4. A system for determining features of consecutive frames of images, characterized in that, The continuous frame image feature determination system includes: The acquisition module is used to acquire continuous frame image data of the feature to be determined, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data of the feature to be determined. The reference data selection module is used to determine the temporal order of each frame of image data according to the time point, and to determine the earliest data according to the temporal order, which is the first frame of image data, and select the first frame of image data as reference data. The first determining module is used to map the reference data to the coordinate system where the continuous frame image data is located for calibration, so as to obtain the reference coordinates of the reference data at the corresponding time point; The second determining module is used to determine the offset angle based on the reference coordinates; determine the time interval between consecutive frames based on the preset frame rate; and determine the horizontal and vertical displacement of the second frame image data based on the offset angle, real-time relative vehicle speed, reference coordinates, time interval, and the time point corresponding to each frame image data in the consecutive frame image data. By combining the horizontal and vertical displacement offsets, the predicted coordinates of each point in the second frame of image data are obtained. The corresponding offset angles are then determined based on the predicted coordinates and offset parameters of each point in the preceding frame of image data. Finally, the predicted coordinates of each point in the following frame of image data are determined based on the corresponding offset angles, real-time relative vehicle speed, the predicted coordinates of each point in the preceding frame of image data, the preset frame rate, and the time points corresponding to each frame of image data in the continuous frame of image data. The predicted coordinates of each point in the same frame of image data are then connected to form a graph, yielding the features of the next adjacent frame of image data in each continuous frame of image data.

5. A continuous frame image annotation system, characterized in that, The continuous frame image annotation system includes: The acquisition module is used to acquire continuous frame image data to be labeled, the time point corresponding to each frame image data in the continuous frame image data, and the real-time relative speed of the vehicle to be controlled relative to the target object corresponding to the continuous frame image data to be labeled. The reference data selection module is used to determine the temporal order of each frame of image data according to the time point, and to determine the earliest data according to the temporal order, which is the first frame of image data, and to select the first frame of image data as reference data. The first determining module is used to map the reference data to the coordinate system where the continuous frame image data is located for calibration, so as to obtain the reference coordinates of the reference data at the corresponding time point; The second determining module is used to determine the offset angle based on the reference coordinates; determine the time interval between consecutive frames based on the preset frame rate; and determine the horizontal and vertical displacement of the second frame image data based on the offset angle, real-time relative vehicle speed, reference coordinates, time interval, and the time point corresponding to each frame image data in the consecutive frame image data. By combining the horizontal and vertical displacement offsets, the predicted coordinates of each point in the second frame of image data are obtained. The corresponding offset angle is determined sequentially based on the predicted coordinates and offset parameters of each point in the adjacent previous frame of image data. The predicted coordinates of each point in the adjacent next frame of image data are then determined sequentially based on the corresponding offset angle, real-time relative vehicle speed, the predicted coordinates of each point in the adjacent previous frame of image data, the preset frame rate, and the time points corresponding to each frame of image data in the continuous frame of image data. Finally, the predicted coordinates of each point in the same frame of data are connected to form a graph, obtaining the features of the next adjacent frame of data in each continuous frame of image data. The annotation module is used to perform category recognition and annotation on the features of the continuous frame images, and complete the annotation.

6. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the method as described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the method as described in any one of claims 1-3.

Images