A data audit interaction method and related device

Abstract

The present application discloses a data review interaction method and related devices, and relates to the field of data annotation technology. The method includes: obtaining a sequence of annotated data frames, each annotated data frame of the annotated data frame sequence includes a point cloud frame annotating a target and an image frame annotating the target corresponding to the point cloud frame; displaying a sequence of annotated point cloud frames composed of the annotated point cloud frames in the annotated data frame sequence on a first interface; when a first user operation for selecting a target to be reviewed from the annotated point cloud frames displayed on the first interface and triggering an audit is detected, extracting a subsequence of annotated data frames corresponding to the target to be reviewed from the annotated data frame sequence; and displaying the extracted subsequence of annotated data frames on a second interface so that the user can review the annotation information of the target to be reviewed and modify it when the audited annotation information is incorrect. The data review interaction method disclosed in the present application can help users quickly and effectively complete the review and modification of annotation information.

Description

Technical Field

[0001] The present application relates to the technical field of data annotation, and in particular to a data review interaction method and related devices. Background Art

[0002] In some fields (such as intelligent driving), it is necessary to annotate the data frame sequences obtained by radar, lidar and cameras (annotate the three-dimensional bounding box, logo, type, etc. of the target in each data frame).

[0003] At present, the process of labeling the data frame sequence acquired based on the lidar and camera is as follows: each data frame in the data frame sequence (each data frame includes a point cloud frame and an image frame corresponding to the point cloud frame) is pre-labeled through a tracking algorithm. Since the labeling results obtained by labeling each data frame in the data frame sequence through the tracking algorithm are often not accurate enough, it is usually necessary to manually review the pre-labeling results obtained through the tracking algorithm and make modifications when there are problems with the labeling information.

[0004] However, the number of data frames is often large, and there is often more than one target in each data frame. How to help users quickly and effectively complete the review and modification of pre-labeling results is a problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, the present application provides a data review interaction method and related devices to help users quickly and effectively complete the review and modification of pre-annotated results. The technical solution is as follows:

[0006] The first aspect of the present application provides a data review interaction method, comprising:

[0007] Acquire a sequence of labeled data frames, wherein the sequence of labeled data frames is a sequence of data frames after each data frame of an unlabeled data frame sequence is pre-labeled, and each labeled data frame of the sequence of labeled data frames includes a point cloud frame with a target labeled and an image frame with a target labeled corresponding to the point cloud frame;

[0008] Displaying the labeled point cloud frames of the labeled point cloud frame sequence on the first interface, wherein the labeled point cloud frame sequence is composed of the labeled point cloud frames in the labeled data frame sequence;

[0009] When a first user operation for selecting a target to be reviewed from the annotated point cloud frames displayed on the first interface and triggering review is detected, a subsequence of annotated data frames corresponding to the target to be reviewed is extracted from the annotated data frame sequence, wherein each annotated data frame of the subsequence of annotated data frames is a data frame annotating the target to be reviewed;

[0010] The marked data frame subsequence corresponding to the target to be reviewed is displayed on the second interface, so that the user can review the marked information of the target to be reviewed and modify it when there is a problem with the marked information of the target to be reviewed.

[0011] In a possible implementation, extracting a subsequence of labeled data frames corresponding to the target to be reviewed from the labeled data frame sequence includes:

[0012] Finding the data frame in which the target to be reviewed is marked from the marked data frame sequence to obtain a first marked data frame subsequence;

[0013] Determining whether the first marked data frame subsequence has discontinuous frame numbers;

[0014] If the first marked data frame subsequence has discontinuous frame numbers, then fill in the missing data frames between the marked data frames with discontinuous frame numbers, and mark the position of the target to be reviewed in the filled-in data frames to obtain a second marked data frame subsequence;

[0015] According to a preset data extraction range, a data region containing the target to be reviewed is extracted from each labeled data frame of the second labeled data frame subsequence to obtain a labeled data frame subsequence corresponding to the target to be reviewed.

[0016] In a possible implementation, marking the position of the to-be-audited target in the supplemented data frame includes:

[0017] Predicting the position of the target to be checked in the supplemented data frame according to the motion state information of the target to be checked in the forward adjacent data frame of the supplemented data frame and the motion state information of the target to be checked in the backward adjacent data frame of the supplemented data frame;

[0018] A three-dimensional bounding box representing the location of the object to be reviewed is added to the supplemented data frame.

[0019] In a possible implementation, displaying the labeled data frame subsequence corresponding to the to-be-audited target on the second interface includes:

[0020] A batch loading method based on multithreading is adopted to load the labeled data frame subsequences corresponding to the target to be reviewed in batches to the second interface for display.

[0021] In a possible implementation, the data audit interaction method further includes:

[0022] If the user modifies the annotation information of the target to be reviewed, the modification result of the annotation information of the target to be reviewed is updated in the first interface.

[0023] In a possible implementation, the data review interaction method further includes: when a second user operation for triggering marking of the review status of the target to be reviewed is detected, marking the review status of the target to be reviewed as reviewed.

[0024] In a possible implementation, the data review interaction method further includes: when a third user operation for triggering motion trajectory generation is detected, converting each point cloud frame in the sequence of annotated data frames with the latest annotation information into a global coordinate system to obtain each point cloud frame in the global coordinate system;

[0025] Merging each point cloud frame in the global coordinate system into a global point cloud image;

[0026] The global point cloud map is displayed on the third interface, and the motion trajectory of the designated target is rendered in the global point cloud map according to the latest annotation information of the designated target, so that the user can judge whether the latest annotation information of the designated target is incorrect and whether the designated target is missing according to the continuity and curvature change of the motion trajectory of the designated target.

[0027] In a possible implementation, fusing the point cloud frames in the global coordinate system into a global point cloud image includes:

[0028] For each point cloud frame in the global coordinate system: filtering out noise points and outliers in the point cloud frame to obtain a filtered point cloud frame; performing voxel filtering on the filtered point cloud frame according to a preset voxel size to obtain a voxel filtered point cloud frame;

[0029] After obtaining the point cloud frames after filtering of each voxel, the point cloud frames after filtering of each voxel are superimposed to obtain a global point cloud map.

[0030] In a possible implementation, rendering the motion trajectory of the designated target in the global point cloud image according to the latest annotation information of the designated target includes:

[0031] The starting position, end position and designated position of the designated target are rendered in the global point cloud map according to the latest annotation information of the designated target, and the motion trajectory of the designated target from the starting position to the designated position and the motion trajectory of the designated target from the designated position to the end position are rendered according to the latest annotation information of the designated target, wherein the designated position is the position of the designated target in the designated point cloud frame.

[0032] A second aspect of the present application provides a data audit interaction device, comprising: a data acquisition module, a first data display module, a data extraction module, and a second data display module;

[0033] The data acquisition module is used to acquire a sequence of labeled data frames, wherein the sequence of labeled data frames is a sequence of data frames after each data frame of an unlabeled data frame sequence is pre-labeled, and each labeled data frame of the sequence of labeled data frames includes a point cloud frame with a target labeled and an image frame with a target labeled corresponding to the point cloud frame;

[0034] The first data display module is used to display the labeled point cloud frames of the labeled point cloud frame sequence on the first interface, wherein the labeled point cloud frame sequence is composed of the labeled point cloud frames in the labeled data frame sequence;

[0035] The data extraction module is configured to extract a subsequence of labeled data frames corresponding to the target to be reviewed from the labeled data frame sequence when a first user operation for selecting a target to be reviewed from the labeled point cloud frame displayed on the first interface and triggering review is detected, wherein each labeled data frame of the subsequence of labeled data frames is a data frame that labels the target to be reviewed;

[0036] The second data display module is used to display the labeled data frame subsequence corresponding to the target to be reviewed on the second interface, so that the user can review the labeled information of the target to be reviewed and modify it if there is any problem with the labeled information of the target to be reviewed.

[0037] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0038] The memory is used to store computer programs;

[0039] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the data review interaction methods described above.

[0040] The fourth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned data review interaction methods.

[0041] A fifth aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the steps of any one of the above-described data review interaction methods.

[0042] With the help of the above technical scheme, the data review interaction method provided by the present application first obtains a labeled data frame sequence, and then displays the labeled point cloud frames of the labeled point cloud frame sequence (a sequence composed of each labeled point cloud frame in the labeled data frame sequence) on the first interface. When a first user operation for selecting a target to be reviewed from the labeled point cloud frames displayed on the first interface and triggering the review is detected, a labeled data frame subsequence corresponding to the target to be reviewed is extracted from the labeled data frame sequence, and then the labeled data frame subsequence corresponding to the target to be reviewed is displayed on the second interface, so that the user can review the labeling information of the target to be reviewed and modify it if the labeling information of the target to be reviewed is incorrect. The data review interaction method provided in the present application can extract a labeled data frame subsequence corresponding to the target selected by the user from the first interface from the labeled data frame sequence, and can display the extracted labeled data frame subsequence in the second interface. Since the second interface displays all data frames labeled with the target selected by the user, the user can easily and quickly review the labeling information of the selected target. Since only one target is selected for review each time, there will be no review confusion. It can be seen that the data review interaction method provided in the present application can help users quickly and effectively complete the review and modification of the labeled information. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0045] Figure 2 A flowchart of a data review interaction method provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of displaying annotated point cloud frames of an annotated point cloud frame sequence on a first interface provided by an embodiment of the present application;

[0047] Figure 4 A schematic diagram of displaying a marked data frame subsequence corresponding to a to-be-audited target on a second interface provided in an embodiment of the present application;

[0048] Figure 5A schematic diagram of a process for extracting a subsequence of labeled data frames corresponding to a target to be reviewed from a sequence of labeled data frames when a first user operation for selecting a target to be reviewed from a labeled point cloud frame displayed on a first interface and triggering review is detected according to an embodiment of the present application;

[0049] Figure 6 A schematic diagram of a point cloud frame after voxel filtering provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of a global point cloud image obtained by superimposing point cloud frames after filtering of each voxel provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of rendering the motion trajectory of a specified target in a global point cloud map provided by an embodiment of the present application;

[0052] Fig. 9 A schematic diagram of the structure of a data review interaction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0054] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0055] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0056] The data audit interaction method provided in the present application can be applied to a terminal. The terminal can be a mobile phone, a tablet computer, a wearable device, a robot, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.

[0057] Figure 1 An optional hardware structure diagram of a terminal is shown.

[0058] refer to Figure 1 As shown, the terminal may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 1 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0059] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0060] The display unit 140 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0061] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0062] The processor 170 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0063] Among them, the memory 120 can be used to store software codes related to the data review interaction method, the processor 170 can execute the software codes in the memory 120, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0064] The radio frequency unit 110 (optional) can be used to receive and send information or receive and send signals during a call. For example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 110 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0065] In the embodiment of the present application, the radio frequency unit 110 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 110 is optional, and can be replaced by other communication interfaces, such as a network port.

[0066] The terminal also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0067] The terminal also includes an external interface 180, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0068] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0069] In the process of implementing this case, it was found that there are currently some auxiliary review tools that assist users (annotation reviewers) in reviewing the annotation results of data frame sequences. The existing auxiliary review tools can realize the display and switching of data frames. When users review the annotation results of data frame sequences, they review the annotation results frame by frame, that is, review whether the annotation information of each target in each data frame is correct. For some scenes (such as overtaking and U-turn scenes), users need to constantly cut frames back and forth to see if they are the same target, and it is easy to get confused when reviewing the annotation information of each target in the same data frame at the same time. It can be seen that the above-mentioned review method has low review efficiency and the data quality cannot be guaranteed. In response to the above problems, research was conducted. Through continuous research, a data review interaction method with better effect was finally proposed. This data review interaction method can greatly improve the user's review efficiency for annotation information and improve data quality.

[0070] Next, the data review interaction method provided by this application is introduced through the following embodiments.

[0071] See also Figure 2 , shows a schematic diagram of the process of the data review interaction method provided in an embodiment of the present application, which may include:

[0072] Step S201: Acquire a sequence of labeled data frames.

[0073] The labeled data frame sequence is a data frame sequence obtained by pre-labeling each data frame of the unlabeled data frame sequence using a tracking algorithm, that is, the labeled data frame sequence is a data frame sequence with labeling information.

[0074] It should be noted that the unlabeled data frame sequence is obtained based on the lidar and the camera, and each data frame in the unlabeled data frame sequence includes a point cloud frame and an image frame corresponding to the point cloud frame. Correspondingly, each labeled data frame in the labeled data frame sequence includes a point cloud frame with the target labeled and an image frame with the target labeled corresponding to the point cloud frame.

[0075] Step S202: displaying the labeled point cloud frames of the labeled point cloud frame sequence on the first interface.

[0076] The labeled point cloud frame sequence is a sequence composed of the labeled point cloud frames in the labeled data frame sequence. The labeled point cloud frame sequence can be obtained, parsed into a readable data format, and rendered to the first interface for display.

[0077] After obtaining the labeled data frame sequence, in a possible implementation, all the labeled point cloud frames contained in the labeled point cloud frame sequence can be displayed on the first interface. Considering that the number of labeled point cloud frames contained in the labeled point cloud frame sequence is often large, it is not convenient for users to view all the labeled point cloud frames. In order to facilitate users to view, in another possible implementation, such as Figure 3 As shown, a labeled point cloud frame of the labeled point cloud frame sequence can be displayed in the first area of ​​the first interface, and a progress bar can be displayed in the second area of ​​the first interface. The user can switch the labeled point cloud frame displayed in the first area by operating the progress bar in the second area, that is, the user can display any labeled point cloud frame of the labeled point cloud frame sequence in the first area by operating the progress bar in the second area. It should be noted that the annotation information of any labeled point cloud frame may include the identification (trackID), three-dimensional bounding box, type and other information of the target marked in the point cloud frame, such as Figure 3 As shown, when the labeled point cloud frame is displayed in the first area of ​​the first interface, the point cloud frame with a three-dimensional bounding box and an identifier (trackID) can be displayed.

[0078] Step S203: when a first user operation for selecting a target to be reviewed from the labeled point cloud frame displayed on the first interface and triggering review is detected, a labeled data frame subsequence corresponding to the target to be reviewed is extracted from the labeled data frame sequence.

[0079] Each of the labeled data frames in the labeled data frame subsequence is a data frame that marks a target to be reviewed.

[0080] The user can select a target to be reviewed from the labeled point cloud frame displayed on the first interface and trigger a review of the selected target (for example, the user can double-click a labeled target in the labeled point cloud frame displayed on the first interface, and for another example, the user can first click to select a labeled target in the labeled point cloud frame displayed on the first interface, and then trigger a review of the selected target through a shortcut). When a first user operation for selecting a target to be reviewed from the labeled point cloud frame displayed on the first interface and triggering a review is detected, a labeled data frame subsequence corresponding to the target to be reviewed is extracted from the labeled data frame sequence.

[0081] When extracting the labeled data frame subsequence corresponding to the target to be reviewed from the labeled data frame sequence, the labeled data frame subsequence corresponding to the target to be reviewed may be extracted from the labeled data frame sequence according to the identifier (trackID) of the target to be reviewed.

[0082] Step S204: Displaying the annotated data frame subsequence corresponding to the target to be reviewed on the second interface, so that the user can review the annotated information of the target to be reviewed and modify it if there is any problem with the annotated information of the target to be reviewed.

[0083] In one possible implementation, when displaying the labeled data frame subsequence corresponding to the target to be reviewed, all the labeled data frames can be loaded together to the second interface for display. Considering that the data amount of the labeled data frame subsequence corresponding to the target to be reviewed is large, in order to avoid jamming and improve interaction efficiency, in another possible implementation, a multi-threaded batch loading method can be adopted to load the labeled data frame subsequences corresponding to the target to be reviewed in batches to the second interface for display. For example, the 1st to 6th labeled data frames are first loaded based on thread 1. After the 1st to 6th labeled data frames are loaded, the 7th to 12th labeled data frames are loaded based on thread 2, and so on, until all the labeled data frames are loaded.

[0084] In this embodiment, the first interface and the second interface are homologous multi-window page communications, and the second interface can be added through window.open.

[0085] See also Figure 4 , shows a schematic diagram of displaying the labeled data frame subsequence corresponding to the target to be reviewed on the second interface. After the labeled data frame subsequence corresponding to the target to be reviewed is displayed on the second interface, the user can review the labeling information of the target to be reviewed in each data frame of the labeled data frame subsequence corresponding to the target to be reviewed. When reviewing, the user can check whether the target framed by the three-dimensional bounding box of the target to be reviewed in each data frame is the same target, and can also check whether there is indeed a target in the three-dimensional bounding box of the target to be reviewed. If it is found through review that the labeling information (such as trackID, type, etc.) of the target to be reviewed in some data frames is incorrect, the user can modify the incorrect labeling information in batches. In addition to reviewing the above two aspects, the user can also review the fit of the three-dimensional bounding box of the target to be reviewed in the corresponding point cloud frame and the target in the box according to the fit of the bounding box of the target to be reviewed in the image frame and the target in the box. If the three-dimensional bounding box of the target to be reviewed in the point cloud frame does not fit the target in the box well, the three-dimensional bounding box of the target to be reviewed is adjusted to fit the target in the box. When the user modifies the annotation information of the target to be reviewed, the modification result of the annotation information of the target to be reviewed can be updated to the first interface.

[0086] After reviewing the annotation information of the target to be reviewed, the user can mark the review status of the target to be reviewed as reviewed, that is, when a second user operation for triggering marking the review status of the target to be reviewed is detected, the review status of the target to be reviewed is marked as reviewed.

[0087] After completing the review of the selected target, the user can select the next target to be reviewed on the first interface and trigger the review, that is, repeat step S203 to step S204 until the review of all targets in the labeled data frame sequence is completed.

[0088] The data review interaction method provided in the embodiment of the present application first obtains a labeled data frame sequence, and then displays the labeled point cloud frames of the labeled point cloud frame sequence (a sequence composed of each labeled point cloud frame in the labeled data frame sequence) on a first interface. When a first user operation for selecting a target to be reviewed from the labeled point cloud frames displayed on the first interface and triggering a review is detected, a labeled data frame subsequence corresponding to the target to be reviewed is extracted from the labeled data frame sequence, and then the labeled data frame subsequence corresponding to the target to be reviewed is displayed on the second interface, so that the user can review the labeling information of the target to be reviewed and modify it if the labeling information of the target to be reviewed is incorrect. The data review interaction method provided in the embodiment of the present application can extract a labeled data frame subsequence corresponding to the target selected by the user from the first interface from the labeled data frame sequence, and can display the extracted labeled data frame subsequence in the second interface. Since the second interface displays all data frames labeled with the target selected by the user, the user can easily and quickly review the labeling information of the selected target. Since only one target is selected for review each time, there will be no review confusion. It can be seen that the data review interaction method provided in the embodiment of the present application can help users quickly and effectively complete the review and modification of the labeled information.

[0089] In another embodiment of the present application, the specific implementation process of "Step S203: when a first user operation for selecting a target to be reviewed from the labeled point cloud frame displayed on the first interface and triggering a review is detected, a labeled data frame subsequence corresponding to the target to be reviewed is extracted from the labeled data frame sequence" in the above embodiment is introduced.

[0090] like Figure 5 As shown, when a first user operation for selecting a target to be reviewed from the labeled point cloud frame displayed on the first interface and triggering review is detected, the process of extracting a labeled data frame subsequence corresponding to the target to be reviewed from the labeled data frame sequence may include:

[0091] Step S501: When a first user operation for selecting a target to be reviewed from the labeled point cloud frame displayed on the first interface and triggering review is detected, a data frame labeled with the target to be reviewed is found from the labeled data frame sequence to obtain a first labeled data frame subsequence.

[0092] According to the identifier (trackID) of the target to be reviewed selected by the user, a data frame in which the target to be reviewed selected by the user is marked can be found from the marked data frame sequence to obtain a first marked data frame subsequence.

[0093] Exemplarily, if the target to be reviewed selected by the user has trackID=2, a data frame containing the target with trackID=2 can be found from the labeled data frame sequence, and the found labeled data frames form a first labeled data frame subsequence.

[0094] Step S502: Determine whether the first marked data frame subsequence has discontinuous frame numbers.

[0095] Exemplarily, the first labeled data frame subsequence includes 6 labeled data frames, and the frame numbers of the 6 labeled data frames are 3, 4, 6, 7, 8, and 9 respectively. Since the frame numbers of the second labeled data frame and the third labeled data frame are not continuous, it can be determined that the first labeled data frame subsequence has discontinuous frame numbers.

[0096] Step S503-a1: If the first marked data frame subsequence has discontinuous frame numbers, the missing data frames are supplemented between the marked data frames with discontinuous frame numbers, and the positions of the targets to be reviewed are marked in the supplemented data frames to obtain a second marked data frame subsequence.

[0097] After filling in the missing data frames between the labeled data frames with discontinuous frame numbers, the position of the target to be reviewed in the filled-in data frame can be predicted based on the motion status information of the target to be reviewed in the forward adjacent data frame of the filled-in data frame and the motion status information of the target to be reviewed in the backward adjacent data frame of the filled-in data frame, and then, a three-dimensional bounding box representing the position of the target to be reviewed is added to the filled-in data frame.

[0098] The motion state information of the target to be reviewed may include, but is not limited to, part or all of the following information: the position of the target to be reviewed, the heading angle of the target to be reviewed, the speed of the target to be reviewed, the acceleration of the target to be reviewed, and the like.

[0099] Exemplarily, the first labeled data frame subsequence includes 6 labeled data frames, and the frame numbers of the 6 labeled data frames are 3, 4, 6, 7, 8, and 9, respectively. Since the frame numbers of the second labeled data frame and the third labeled data frame are not continuous, the labeled data frame with frame number 5 is obtained from the labeled data frame sequence and inserted between the second labeled data frame and the third labeled data frame of the first labeled data frame subsequence. The position of the target to be reviewed in the data frame with frame number 5 is predicted based on the motion state information of the target to be reviewed in the data frame with frame number 4 and the motion state information of the target to be reviewed in the data frame with frame number 6. Then, a three-dimensional bounding box representing the position of the target to be reviewed is added to the data frame with frame number 5, and a second labeled data frame subsequence including labeled data frames with frame numbers 3, 4, 5, 6, 7, 8, and 9 is obtained.

[0100] In one possible implementation, the process of predicting the position of the target to be reviewed in the supplemented data frame based on the motion status information of the target to be reviewed in the forward adjacent data frame of the supplemented data frame and the motion status information of the target to be reviewed in the backward adjacent data frame of the supplemented data frame may include: substituting the motion status information of the target to be reviewed in the forward adjacent data frame of the supplemented data frame and the motion status information of the target to be reviewed in the backward adjacent data frame of the supplemented data frame as boundary conditions into the curve equation for solving, calculating the curve correlation coefficient, and inferring the position of the target to be reviewed in the supplemented data frame based on the curve equation coefficient and the time interval between adjacent data frames.

[0101] Step S503 - a2 : according to a preset data extraction range, extract a data region containing the target to be reviewed from each labeled data frame of the second labeled data frame subsequence to obtain a labeled data frame subsequence corresponding to the target to be reviewed.

[0102] The marked data frame subsequence corresponding to the target to be reviewed may be a marked data frame subsequence in which only the target to be reviewed is marked.

[0103] In order to reduce the amount of data and at the same time, to reduce audit interference, after obtaining the second labeled data frame subsequence, this embodiment extracts the data area containing the target to be audited from each labeled data frame of the second labeled data frame subsequence according to a preset data extraction range, and the labeled data frame subsequence corresponding to the target to be audited is composed of the data areas extracted from each labeled data frame of the second labeled data frame subsequence.

[0104] After obtaining the labeled data frame subsequence corresponding to the target to be reviewed, the multi-threaded batch loading method can be adopted to load the labeled data frame subsequence corresponding to the target to be reviewed in batches to the second interface for display. Then, the user can review the annotation information of each labeled data frame of the labeled data frame subsequence corresponding to the target to be reviewed. During the review, it will be reviewed whether there is a target in the above-mentioned newly added three-dimensional bounding box. If there is no point cloud or the point cloud is sparse in a three-dimensional bounding box, it can be determined that there is no target in the three-dimensional bounding box, that is, the three-dimensional bounding box is a misidentified three-dimensional bounding box and can be deleted.

[0105] Step S503-b: If there is no discontinuous frame number in the first labeled data frame subsequence, then according to the preset data extraction range, the data area containing the target to be reviewed is extracted from each labeled data frame of the first labeled data frame subsequence to obtain the labeled data frame subsequence corresponding to the target to be reviewed.

[0106] If there is no discontinuous frame number in the first marked data frame subsequence, the data region containing the target to be reviewed is directly extracted from each marked data frame of the first marked data frame subsequence to obtain the marked data frame subsequence corresponding to the target to be reviewed.

[0107] In order to further improve the accuracy of the annotation information, in another embodiment of the present application, after completing the review and modification of the annotation information of each target in the annotated data frame sequence in the manner of steps S201 to S204, the following process may be further performed:

[0108] Step a1: when a third user operation for triggering motion trajectory generation is detected, each point cloud frame in the annotated data frame sequence with the latest annotation information is converted to a global coordinate system to obtain each point cloud frame in the global coordinate system.

[0109] The user can operate in the first interface to trigger the generation of motion trajectories. When a third user operation for triggering the generation of motion trajectories is detected, each point cloud frame in the labeled data frame sequence with the latest annotation information is converted to the global coordinate system to obtain each point cloud frame in the global coordinate system.

[0110] It should be noted that after the laser radar is installed and calibrated, the conversion relationship between the laser radar coordinate system and the global coordinate system will be fixed. This embodiment converts each point cloud frame to the global coordinate system based on the known conversion relationship between the laser radar coordinate system and the global coordinate system to obtain each point cloud frame in the global coordinate system.

[0111] Step a2: fuse each point cloud frame in the global coordinate system into a global point cloud map.

[0112] In a possible implementation, each point cloud frame in the global coordinate system may be directly superimposed to obtain a global point cloud image.

[0113] Considering that the point cloud of the global point cloud obtained by directly superimposing the point cloud frames in the global coordinate system is relatively dense, in order to reduce the amount of data and make the rendered image outline clearer, in another possible implementation method, for each point cloud frame in the global coordinate system, the noise points and outliers of the point cloud frame can be filtered out first to obtain a filtered point cloud frame, and then a voxel filter (such as a VoxelGrid filter) is used to perform voxel filtering on the filtered point cloud frame according to a preset voxel size to obtain a voxel filtered point cloud frame. After obtaining each voxel filtered point cloud frame, the voxel filtered point cloud frames are superimposed to obtain a global point cloud map. Please refer to Figure 6 , shows an example of a voxel-filtered point cloud frame, see Figure 7 , shows an example of a global point cloud image obtained by superimposing point cloud frames after filtering of each voxel.

[0114] Step a3, display the global point cloud map on the third interface, and render the motion trajectory of the specified target in the global point cloud map according to the latest annotation information of the specified target, so that the user can judge whether the latest annotation information of the specified target is incorrect and whether there is any missing label of the specified target according to the continuity and curvature change of the motion trajectory of the specified target.

[0115] When the user operates on the first interface to trigger the generation of the motion trajectory, the user may select a target from the point cloud frame currently displayed on the first interface, or may not select a target. If the user selects a target from the point cloud frame currently displayed on the first interface (one target or multiple targets may be selected), the designated target in step a3 is the target selected by the user. If the user does not select a target from the point cloud frame currently displayed on the first interface, the designated target in step a3 may be the first N (for example, 3) targets in the point cloud frame currently displayed on the first interface.

[0116] There are many ways to render the motion trajectory of a specified target in a global point cloud map based on the latest annotation information of the specified target. In one possible implementation method, the starting point and end point positions of the specified target can be rendered based on the latest annotation information of the specified target (such as the center point position of each latest three-dimensional bounding box of the specified target, each latest three-dimensional bounding box of the specified target refers to the latest three-dimensional bounding box that represents the position of the specified target in each point cloud frame where the specified target is located), and the motion trajectory between the starting point and the end point positions can be rendered.

[0117] In another possible implementation, the starting point, end point and designated position of the designated target can be rendered in the global point cloud map according to the latest annotation information of the designated target (such as the center point position of each latest 3D bounding box of the designated target), and the motion trajectory of the designated target from the starting point to the designated position and the motion trajectory of the designated target from the designated position to the end point can be rendered according to the latest annotation information of the designated target (such as the center point position of each latest 3D bounding box of the designated target). The designated position is the position of the designated target in the designated point cloud frame. Please refer to Figure 8 , showing a schematic diagram of a starting position, an end position, a designated position, a motion trajectory of the designated target from the starting position to the designated position, and a motion trajectory of the designated target from the designated position to the end position.

[0118] For example, the first interface currently displays the mth point cloud frame of the annotated point cloud frame sequence. When the user selects a target in the point cloud frame currently displayed on the first interface and triggers the motion trajectory generation, a global point cloud map is generated, and the starting point position P of the target selected by the user is rendered in the global point cloud map according to the center point position of each latest three-dimensional bounding box of the target selected by the user. s , the position P of the target selected by the user in the mth point cloud framem And the stop position P of the target selected by the user e , and render the target selected by the user from the starting position P according to the center point position of each latest three-dimensional bounding box of the target selected by the user s To P m The motion trajectory and the target selected by the user are obtained from P m The motion trajectory to the dead point position Pe.

[0119] After rendering the motion trajectory of the specified target in the global point cloud map, the user can judge whether the latest annotation information (such as trackID) of the specified target is correct and whether there is any missing label of the specified target according to the continuity and curvature change of the motion trajectory of the specified target (missing label means that there are specified targets in some data frames but they are not marked). If it is determined that there is a problem with the annotation information (such as trackID) of the specified target according to the motion trajectory of the specified target, the user can modify and improve the annotation information of the specified target.

[0120] After the user completes further review of the annotation information of the designated target according to the motion trajectory of the designated target, the user can continue to select targets and trigger the generation of the trajectory of the selected targets, and then further review the annotation information of the selected targets according to the generated motion trajectory.

[0121] The above mainly introduces the data review interaction method provided in the embodiment of the present application, and the device corresponding to the above data review interaction method will be introduced below.

[0122] See also Fig. 9 , Fig. 9 This is a structural diagram of a data review interaction device provided in an embodiment of the present application. The data review interaction device may include: a data acquisition module 901, a first data display module 902, a data extraction module 903 and a second data display module 904.

[0123] The data acquisition module 901 is used to acquire a sequence of labeled data frames.

[0124] The labeled data frame sequence is a data frame sequence obtained by pre-labeling each data frame in the unlabeled data frame sequence, and each labeled data frame in the labeled data frame sequence includes a point cloud frame with a target labeled and an image frame with the target labeled corresponding to the point cloud frame.

[0125] The first data display module 902 is used to display the labeled point cloud frames of the labeled point cloud frame sequence on a first interface.

[0126] The labeled point cloud frame sequence is composed of the labeled point cloud frames in the labeled data frame sequence.

[0127] The data extraction module 903 is used to extract a subsequence of labeled data frames corresponding to the target to be reviewed from the labeled data frame sequence when a first user operation for selecting a target to be reviewed from the labeled point cloud frame displayed on the first interface and triggering review is detected.

[0128] Each of the labeled data frames in the labeled data frame subsequence is a data frame that marks a target to be reviewed.

[0129] The second data display module 904 is used to display the annotated data frame subsequence corresponding to the target to be reviewed on the second interface, so that the user can review the annotated information of the target to be reviewed and modify it if there is any problem with the annotated information of the target to be reviewed.

[0130] In a possible implementation, when the data extraction module 903 extracts the labeled data frame subsequence corresponding to the target to be reviewed from the labeled data frame sequence, it is specifically used to:

[0131] Find the data frame that marks the target to be reviewed from the marked data frame sequence to obtain a first marked data frame subsequence;

[0132] Determine whether the frame numbers of the first marked data frame subsequence are discontinuous;

[0133] If the first marked data frame subsequence has discontinuous frame numbers, then fill in the missing data frames between the marked data frames with discontinuous frame numbers, and mark the position of the target to be reviewed in the filled-in data frames to obtain a second marked data frame subsequence;

[0134] According to the preset data extraction range, a data region containing the target to be reviewed is extracted from each labeled data frame of the second labeled data frame subsequence to obtain a labeled data frame subsequence corresponding to the target to be reviewed.

[0135] In a possible implementation, when the data extraction module 903 marks the position of the target to be reviewed in the supplemented data frame, it is specifically used to:

[0136] Predicting the position of the target to be checked in the supplemented data frame according to the motion state information of the target to be checked in the forward adjacent data frame of the supplemented data frame and the motion state information of the target to be checked in the backward adjacent data frame of the supplemented data frame;

[0137] A three-dimensional bounding box representing the location of the target to be reviewed is added to the inserted data frame.

[0138] In a possible implementation, when the second data display module 904 displays the labeled data frame subsequence corresponding to the target to be reviewed on the second interface, it is specifically used to:

[0139] A multi-threaded batch loading method is adopted to load the labeled data frame subsequences corresponding to the targets to be reviewed in batches to the second interface for display.

[0140] In a possible implementation, the data audit interaction device may further include: a marking modification result updating module.

[0141] The annotation modification result updating module is used to update the modification result of the annotation information of the target to be reviewed to the first interface when the user modifies the annotation information of the target to be reviewed.

[0142] In a possible implementation, the data review interaction device may further include: a review status marking module.

[0143] The audit status marking module is used to mark the audit status of the target to be audited as audited when a second user operation for triggering marking of the audit status of the target to be audited is detected.

[0144] In a possible implementation, the data review interaction device may further include: a point cloud frame transformation module, a point cloud frame fusion module, a third data display module and a motion trajectory rendering module.

[0145] The point cloud frame transformation module is used to transform each point cloud frame in the annotated data frame sequence with the latest annotation information into the global coordinate system to obtain each point cloud frame in the global coordinate system when a third user operation for triggering the generation of the motion trajectory is detected.

[0146] The point cloud frame fusion module is used to fuse each point cloud frame in the global coordinate system into a global point cloud map.

[0147] The third data display module is used to display the global point cloud map on the third interface.

[0148] The motion trajectory rendering module is used to render the motion trajectory of the specified target in the global point cloud map according to the latest annotation information of the specified target, so that the user can judge whether the latest annotation information of the specified target is incorrect and whether there is any missing label of the specified target based on the continuity and curvature change of the motion trajectory of the specified target.

[0149] In a possible implementation, when the point cloud frame fusion module fuses the point cloud frames in the global coordinate system into a global point cloud image, it is specifically used to:

[0150] For each point cloud frame in the global coordinate system: filter out noise points and outliers in the point cloud frame to obtain a filtered point cloud frame; perform voxel filtering on the filtered point cloud frame according to a preset voxel size to obtain a voxel filtered point cloud frame;

[0151] After obtaining the point cloud frames after filtering of each voxel, the point cloud frames after filtering of each voxel are superimposed to obtain a global point cloud map.

[0152] In a possible implementation, when the motion trajectory rendering module renders the motion trajectory of the designated target in the global point cloud image according to the latest annotation information of the designated target, it is specifically used to:

[0153] The starting position, end position and designated position of the designated target are rendered in the global point cloud map according to the latest annotation information of the designated target, and the motion trajectory of the designated target from the starting position to the designated position and the motion trajectory of the designated target from the designated position to the end position are rendered according to the latest annotation information of the designated target, wherein the designated position is the position of the designated target in the specified point cloud frame.

[0154] The data review interaction device provided in the embodiment of the present application first obtains a labeled data frame sequence, and then displays the labeled point cloud frames of the labeled point cloud frame sequence (a sequence composed of each labeled point cloud frame in the labeled data frame sequence) on a first interface. When a first user operation for selecting a target to be reviewed from the labeled point cloud frames displayed on the first interface and triggering a review is detected, a labeled data frame subsequence corresponding to the target to be reviewed is extracted from the labeled data frame sequence, and then the labeled data frame subsequence corresponding to the target to be reviewed is displayed on the second interface, so that the user can review the labeling information of the target to be reviewed and modify it if the labeling information of the target to be reviewed is incorrect. The data review interaction device provided in the embodiment of the present application can extract a labeled data frame subsequence corresponding to the target selected by the user from the first interface from the labeled data frame sequence, and can display the extracted labeled data frame subsequence in the second interface. Since the second interface displays all data frames labeled with the target selected by the user, the user can easily and quickly review the labeling information of the selected target. Since only one target is selected for review each time, there will be no review confusion. It can be seen that the data review interaction device provided in the embodiment of the present application can help users quickly and effectively complete the review and modification of the labeled information.

[0155] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0156] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0157] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0158] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0159] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement the steps of the data review interaction method provided in the above embodiment.

[0160] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the data review interaction method provided in the above embodiment.

[0161] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the data review interaction method provided in the above embodiment.

[0162] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0163] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0164] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0165] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data audit interaction method, characterized in that: include: Acquire a sequence of labeled data frames, wherein the sequence of labeled data frames is a sequence of data frames after each data frame of an unlabeled data frame sequence is pre-labeled, and each labeled data frame of the sequence of labeled data frames includes a point cloud frame with a target labeled and an image frame with a target labeled corresponding to the point cloud frame; Displaying the labeled point cloud frames of the labeled point cloud frame sequence on the first interface, wherein the labeled point cloud frame sequence is composed of the labeled point cloud frames in the labeled data frame sequence; When a first user operation for selecting a target to be reviewed from the annotated point cloud frames displayed on the first interface and triggering review is detected, a subsequence of annotated data frames corresponding to the target to be reviewed is extracted from the annotated data frame sequence, wherein each annotated data frame of the subsequence of annotated data frames is a data frame annotating the target to be reviewed; Displaying the labeled data frame subsequence corresponding to the target to be reviewed on the second interface, so that the user can review the labeled information of the target to be reviewed and modify it if there is any problem with the labeled information of the target to be reviewed; The data audit interaction method further includes: When a user operation for triggering motion trajectory generation is detected, each point cloud frame in the annotated data frame sequence with the latest annotation information is converted to the global coordinate system to obtain each point cloud frame in the global coordinate system; Merging each point cloud frame in the global coordinate system into a global point cloud image; The global point cloud map is displayed on the third interface, and the motion trajectory of the designated target is rendered in the global point cloud map according to the latest annotation information of the designated target, so that the user can judge whether the latest annotation information of the designated target is incorrect and whether the designated target is missing according to the continuity and curvature change of the motion trajectory of the designated target.

2. The data audit interaction method according to claim 1, characterized in that: The step of extracting a subsequence of labeled data frames corresponding to the target to be reviewed from the labeled data frame sequence includes: Finding the data frame in which the target to be reviewed is marked from the marked data frame sequence to obtain a first marked data frame subsequence; Determining whether the first marked data frame subsequence has discontinuous frame numbers; If the first marked data frame subsequence has discontinuous frame numbers, then fill in the missing data frames between the marked data frames with discontinuous frame numbers, and mark the position of the target to be reviewed in the filled-in data frames to obtain a second marked data frame subsequence; According to a preset data extraction range, a data region containing the target to be reviewed is extracted from each labeled data frame of the second labeled data frame subsequence to obtain a labeled data frame subsequence corresponding to the target to be reviewed.

3. The data audit interaction method according to claim 2, characterized in that: The step of marking the position of the target to be reviewed in the supplemented data frame includes: Predicting the position of the target to be checked in the supplemented data frame according to the motion state information of the target to be checked in the forward adjacent data frame of the supplemented data frame and the motion state information of the target to be checked in the backward adjacent data frame of the supplemented data frame; A three-dimensional bounding box representing the location of the object to be reviewed is added to the supplemented data frame.

4. The data audit interaction method according to claim 1, characterized in that: The displaying of the labeled data frame subsequence corresponding to the target to be reviewed on the second interface includes: A batch loading method based on multithreading is adopted to load the labeled data frame subsequences corresponding to the target to be reviewed in batches to the second interface for display.

5. The data audit interaction method according to claim 1, characterized in that: Also includes: If the user modifies the annotation information of the target to be reviewed, the modification result of the annotation information of the target to be reviewed is updated in the first interface.

6. The data audit interaction method according to claim 1, characterized in that: Also includes: When a second user operation for triggering marking of the review status of the target to be reviewed is detected, the review status of the target to be reviewed is marked as reviewed.

7. The data audit interaction method according to claim 1, characterized in that: The step of fusing the point cloud frames in the global coordinate system into a global point cloud image includes: For each point cloud frame in the global coordinate system: filtering out noise points and outliers in the point cloud frame to obtain a filtered point cloud frame; performing voxel filtering on the filtered point cloud frame according to a preset voxel size to obtain a voxel filtered point cloud frame; After obtaining the point cloud frames after filtering of each voxel, the point cloud frames after filtering of each voxel are superimposed to obtain a global point cloud map.

8. The data audit interaction method according to claim 1, characterized in that: Rendering the motion trajectory of the designated target in the global point cloud map according to the latest annotation information of the designated target includes: The starting position, end position and designated position of the designated target are rendered in the global point cloud map according to the latest annotation information of the designated target, and the motion trajectory of the designated target from the starting position to the designated position and the motion trajectory of the designated target from the designated position to the end position are rendered according to the latest annotation information of the designated target, wherein the designated position is the position of the designated target in the designated point cloud frame.

9. A data audit interaction device, characterized in that: include: A data acquisition module, a first data display module, a data extraction module, and a second data display module; The data acquisition module is used to acquire a sequence of labeled data frames, wherein the sequence of labeled data frames is a sequence of data frames after each data frame of an unlabeled data frame sequence is pre-labeled, and each labeled data frame of the sequence of labeled data frames includes a point cloud frame with a target labeled and an image frame with a target labeled corresponding to the point cloud frame; The first data display module is used to display the labeled point cloud frames of the labeled point cloud frame sequence on the first interface, wherein the labeled point cloud frame sequence is composed of the labeled point cloud frames in the labeled data frame sequence; The data extraction module is configured to extract a subsequence of labeled data frames corresponding to the target to be reviewed from the labeled data frame sequence when a first user operation for selecting a target to be reviewed from the labeled point cloud frame displayed on the first interface and triggering review is detected, wherein each labeled data frame of the subsequence of labeled data frames is a data frame that labels the target to be reviewed; The second data display module is used to display the labeled data frame subsequence corresponding to the target to be reviewed on the second interface, so that the user can review the labeled information of the target to be reviewed and modify it if there is any problem with the labeled information of the target to be reviewed; The data review interaction device also includes: a point cloud frame transformation module, a point cloud frame fusion module, a third data display module and a motion trajectory rendering module; The point cloud frame conversion module is used to convert each point cloud frame in the annotated data frame sequence with the latest annotation information into a global coordinate system to obtain each point cloud frame in the global coordinate system when a user operation for triggering the generation of the motion trajectory is detected; The point cloud frame fusion module is used to fuse the point cloud frames in the global coordinate system into a global point cloud image; The third data display module is used to display the global point cloud image on a third interface; The motion trajectory rendering module is used to render the motion trajectory of the designated target in the global point cloud map according to the latest annotation information of the designated target, so that the user can judge whether the latest annotation information of the designated target is incorrect and whether the designated target is missing according to the continuity and curvature change of the motion trajectory of the designated target.

10. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the data review interaction method as described in any one of claims 1 to 8.

11. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the data review interaction method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the data review interaction method as described in any one of claims 1 to 8.

Images