AI Detection Method and Device for Asynchronous Processing of Video Streams Aligned Based on Timestamps

Abstract

The present invention provides an AI detection method and device for asynchronous processing of video streams based on timestamp alignment, which relates to the technical field of AI detection, and includes: Step 1, a video stream reading thread receives the detection results fed back by an AI detection thread for the previous latest image frame, and associates and sends the current latest image frame and its timestamp to the AI detection thread; Step 2, the AI detection thread performs AI detection on the current latest image frame; Step 3, during the AI detection, the video stream reading thread aligns the image frame and the detection result based on the timestamp carried in the detection result; Step 4, a playback thread displays the aligned image frame and the detection result; Step 5, when the AI detection is completed, Steps 1 to 4 are repeated until a preset stop condition is met. The present invention can effectively alleviate the problems of low rendering performance, significant resource occupation, and latency existing in serial synchronous processing in traditional AI detection technologies.

Description

Technical Field

[0001] The present invention relates to the field of AI detection technology, and in particular to an AI detection method and device based on asynchronous processing of video streams based on timestamp alignment. Background Art

[0002] The traditional AI (Artificial Intelligence) detection method generally obtains the video stream from the drone through OpenCV (OpenSource Computer Vision Library) / FFmpeg (Fast Forward Mpeg) to decode the video and extract frames, then converts the obtained video frames into images of the corresponding format, and then sends the images frame by frame to the AI ​​model for detection, classification and recognition of relevant targets. The detection results output by the AI ​​model are then drawn on each frame of the image using OpenCV, and finally each frame of the image is encoded using FFmpeg to output a video stream in h264 / h265 format.

[0003] However, the above-mentioned AI detection and rendering have certain defects, mainly in that: (1) each step is serially synchronized, and the FPS (Frames Per Second) of the output video stream depends on the inference speed of the AI ​​model, resulting in poor rendering performance; (2) the real-time video frames obtained from the drone camera undergo image transcoding, AI detection, video encoding and other steps, which consumes a lot of time and will produce a large delay, resulting in a large time deviation between the real-time image seen by the rendering terminal and the image at the current moment in the real scene. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an AI detection method and device based on asynchronous processing of video streams with timestamp alignment, which can effectively alleviate the problems of low rendering performance, significant resource occupation and delay in serial synchronous processing in traditional AI detection technology.

[0005] In a first aspect, the present invention provides an AI detection method for asynchronous processing of video streams based on timestamp alignment, the method is applied to a terminal, the terminal is configured with a playback thread, a video stream reading thread and an AI detection thread, the method comprising:

[0006] Step 1: The video stream reading thread receives the detection result fed back by the AI ​​detection thread for the previous latest image frame, and sends the current latest image frame and its timestamp association to the AI ​​detection thread;

[0007] Step 2: The AI ​​detection thread performs AI detection on the latest image frame;

[0008] Step 3: While the AI ​​detection thread is performing AI detection on the current latest image frame, the video stream reading thread aligns the previous latest image frame and the detection result based on the timestamp carried by the detection result, and predicts and aligns the detection results of the image frames between the previous latest image frame and the current latest image frame, and sends the aligned image frames and detection results to the playback thread;

[0009] Step 4, the playback thread displays the aligned image frames and detection results;

[0010] Step 5: When the AI ​​detection thread completes the AI ​​detection for the current latest image frame, repeat steps 1 to 4 until the preset stop condition is met.

[0011] In one implementation, the video stream reading thread is configured with a queue, and the queue stores a map structure corresponding to the image frame; the latest image frame and its timestamp are associated and sent to the AI ​​detection thread, including:

[0012] Construct an initial map structure corresponding to the current latest image frame; wherein the key of the initial map structure is the timestamp of the current latest image frame, the first value is the current latest image frame, and the second value is empty;

[0013] Send the key and first value association of the initial map structure to the AI ​​detection thread.

[0014] In one implementation, aligning the most recent image frame with the detection result based on the timestamp carried by the detection result includes:

[0015] Using the timestamp carried by the detection result as the retrieval condition, retrieve the initial map structure corresponding to the previous latest image frame from the queue;

[0016] Using the detection result, the second value of the initial map structure corresponding to the previous latest image frame is assigned to obtain the target map structure corresponding to the previous latest image frame, so as to align the previous latest image frame with the detection result.

[0017] In one implementation, predicting and aligning detection results of image frames between a previous latest image frame and a current latest image frame includes:

[0018] For each image frame between the previous latest image frame and the current latest image frame, the detection result of the image frame is predicted based on the detection result of the previous adjacent image frame corresponding to the image frame, and the second value of the initial map structure corresponding to the image frame is assigned using the predicted detection result to obtain the target map structure corresponding to the image frame, so as to align the image frame and the predicted detection result.

[0019] In one implementation, sending the aligned image frames and the detection results to the playback thread includes:

[0020] According to the structure entry and exit principle of the queue, the target map structure saved in the queue is sent to the playback thread.

[0021] In one implementation, the playback thread displays the aligned image frames and the detection results, including:

[0022] The playback thread uses OpenCV to read and display the image frames and detection results in the target map structure.

[0023] In one embodiment, the method further includes: while the AI ​​detection thread performs AI detection on the current latest image frame, the video stream reading thread continues to read the next adjacent image frame of the current latest image frame.

[0024] In a second aspect, the present invention further provides an AI detection device for asynchronous processing of video streams based on timestamp alignment, the device is applied to a terminal, the terminal is configured with a playback thread, a video stream reading thread and an AI detection thread, and the device includes:

[0025] The sending and receiving module is used for: the video stream reading thread receives the detection result fed back by the AI ​​detection thread for the previous latest image frame, and sends the current latest image frame and its timestamp to the AI ​​detection thread;

[0026] Image frame detection module, used for: AI detection thread performs AI detection on the latest image frame;

[0027] The image frame and result alignment module is used to: during the period when the AI ​​detection thread performs AI detection on the current latest image frame, the video stream reading thread aligns the previous latest image frame and the detection result based on the timestamp carried by the detection result, and predicts and aligns the detection results of the image frames between the previous latest image frame and the current latest image frame, and sends the aligned image frames and detection results to the playback thread;

[0028] The image frame and result display module is used to: play the thread to display the aligned image frames and detection results;

[0029] The repeated calling module is used to: when the AI ​​detection thread completes the AI ​​detection for the current latest image frame, repeat the sending and receiving module, the image frame detection module, the image frame and result alignment module and the image frame and result display module until the preset stop condition is met.

[0030] In a third aspect, the present invention further provides a terminal, comprising a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement any one of the methods provided in the first aspect.

[0031] In a fourth aspect, the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0032] The present invention provides an AI detection method and device for asynchronous processing of video streams based on timestamp alignment, wherein a video stream reading thread receives a detection result fed back by an AI detection thread for a previous latest image frame, and sends the current latest image frame and its timestamp association to the AI ​​detection thread, so that the AI ​​detection thread performs AI detection on the current latest image frame; while the AI ​​detection thread performs AI detection on the current latest image frame, the video stream reading thread aligns the previous latest image frame and the detection result based on the timestamp carried by the detection result, and predicts and aligns the detection results of image frames between the previous latest image frame and the current latest image frame, and sends the aligned image frames and detection results to a playback thread, so that the playback thread displays the aligned image frames and detection results; when the AI ​​detection thread completes the AI ​​detection of the current latest image frame, the aforementioned process is repeated until a preset stop condition is met. In the above method, the playback thread, video stream reading thread and AI detection thread perform asynchronous processing on the collected image frames, so that the playback thread does not need to wait for the response time of the AI ​​detection thread. When the AI ​​detection thread performs AI detection on the current latest image frame, the playback thread can still play the image frame and detection results in real time, thereby effectively alleviating the problems of low rendering performance, significant resource occupation and delay in serial synchronous processing in traditional AI detection technology. In addition, the alignment between image frames and detection results is achieved by using timestamps, which can solve the problem of misalignment between image frames and detection results during asynchronous processing.

[0033] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A processing flow chart of a traditional AI detection technology provided by an embodiment of the present invention;

[0037] Figure 2 A flowchart of an AI detection method for asynchronous processing of video streams based on timestamp alignment provided by an embodiment of the present invention;

[0038] Figure 3 A technical architecture diagram of an AI detection method for asynchronous processing of video streams based on timestamp alignment provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the structure of an AI detection device for asynchronous processing of video streams based on timestamp alignment provided by an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] See also Figure 1A processing flow chart of a traditional AI detection technology is shown, in which the server obtains the video stream from the drone end using the RTMP (Real-Time Messaging Protocol) / RTSP (Real Time Streaming Protocol) protocol, and performs operations such as pulling streams and extracting frames, AI detection, uploading detection target information / pushing streams to streaming media servers, etc. on the video stream in sequence, and then transmits it to the front end via the HTTP (Hypertext Transfer Protocol) protocol to display detection pictures and live streams. The traditional AI detection technology has the following problems: the FPS of the output video stream depends on the inference speed of the AI ​​model, and the rendering performance is poor; there is a large time deviation between the real-time image seen by the rendering terminal and the image at the current moment in the real scene. Based on this, the present invention implements an AI detection method and device based on asynchronous processing of video streams based on timestamp alignment, which can effectively alleviate the problems of low rendering performance, significant resource occupation and delay in serial synchronous processing in traditional AI detection technology.

[0043] To facilitate understanding of this embodiment, first, an AI detection method for asynchronous processing of video streams based on timestamp alignment disclosed in an embodiment of the present invention is introduced in detail. The method is applied to a terminal, and the terminal is configured with a play thread, a video stream reading thread, and an AI detection thread. Figure 2 The flowchart of an AI detection method for asynchronous processing of video streams based on timestamp alignment is shown, and the method mainly includes the following steps 1 to 5:

[0044] Step 1: The video stream reading thread receives the detection result fed back by the AI ​​detection thread for the previous latest image frame, and sends the current latest image frame and its timestamp association to the AI ​​detection thread.

[0045] The detection result may be a detection frame of the target object and its coordinate information, type information, etc., which is related to actual needs.

[0046] Among them, the current latest image frame can be understood as the latest frame collected by the video stream reading thread when the AI ​​detection thread completes the previous AI detection task, and the previous AI detection task is also the AI ​​detection task performed on the previous latest image frame; and the previous latest image frame is explained in this way.

[0047] Exemplarily, the video stream reading thread disassembles the image frames from the camera video decoding module in real time; when the AI ​​detection task starts, the video stream reading thread sends the first image frame read to the AI ​​detection thread. During the AI ​​detection process of the first image frame, the video stream reading thread continues to disassemble new image frames from the camera video decoding module, such as collecting the second and third image frames; assuming that the video stream reading thread receives the detection result of the first image frame fed back by the AI ​​detection thread when reading the fourth image frame, the video stream reading thread sends the fourth image frame to the AI ​​detection thread. In the above process, the first image frame is the previous latest image frame, and the fourth image frame is the current latest image frame.

[0048] Step 2: The AI ​​detection thread performs AI detection on the latest image frame. In one example, the AI ​​detection thread is configured with an AI model to perform AI detection on the acquired image frame.

[0049] Step 3. While the AI ​​detection thread is performing AI detection on the current latest image frame, the video stream reading thread aligns the previous latest image frame and the detection result based on the timestamp carried by the detection result, and predicts and aligns the detection results of the image frames between the previous latest image frame and the current latest image frame, and sends the aligned image frames and detection results to the playback thread.

[0050] The timestamp is used to record the acquisition time of the image frame.

[0051] In one example, when the video stream reading thread acquires an image frame, it associates and saves the image frame with its timestamp, and sends the image frame with its timestamp to the AI ​​detection thread. On this basis, for the image frame sent to the AI ​​detection thread for processing: when the video stream reading thread receives the detection result of the image frame, the alignment between the image frame and the detection result can be achieved based on the timestamp carried by the detection result.

[0052] In one example, for an image frame that is not sent to the AI ​​detection thread for processing: the detection result of the image frame can be predicted and aligned based on the detection result of the previous adjacent image frame corresponding to the image frame. For example, assuming that the previous latest image frame is the first image frame, the current latest image frame is the fourth image frame, and the second and third image frames are not sent to the AI ​​detection thread for processing, the first image frame is the previous adjacent image frame of the second image frame, and the second image frame is the previous adjacent image frame of the third image frame.

[0053] Step 4: The playback thread displays the aligned image frames and detection results. The playback thread does not need to wait for the response time of the AI ​​detection thread. While the AI ​​detection thread is processing the request, the playback thread can still play the detection image in real time.

[0054] Step 5: When the AI ​​detection thread completes the AI ​​detection for the latest image frame, repeat steps 1 to 4 until a preset stop condition is met, wherein the preset stop condition may be receiving a stop instruction, etc.

[0055] Continuing with the above example, when the AI ​​detection thread completes the AI ​​detection of the 4th image frame, the AI ​​detection result will feed back the detection result of the 4th image frame to the video stream reading thread. Assuming that the video stream reading thread has collected the 5th, 6th, and 7th image frames during the AI ​​detection process, the 4th image frame will be used as the previous latest image frame, and the 7th image frame will be used as the current latest image frame, and the above steps 1 to 4 will be repeated. This cycle is repeated to achieve real-time playback of the detection screen.

[0056] The AI ​​detection method for asynchronous processing of video streams based on timestamp alignment provided in an embodiment of the present invention uses a playback thread, a video stream reading thread and an AI detection thread to perform asynchronous processing on the collected image frames, so that the playback thread does not need to wait for the response time of the AI ​​detection thread. When the AI ​​detection thread performs AI detection on the current latest image frame, the playback thread can still play the image frame and detection results in real time, thereby effectively alleviating the problems of low rendering performance, significant resource occupation and delay in serial synchronous processing in traditional AI detection technology. In addition, the alignment between image frames and detection results is achieved by using timestamps, which can solve the problem of misalignment between image frames and detection results during asynchronous processing.

[0057] For ease of understanding, the embodiments of the present invention provide Figure 3 The technical architecture diagram of an AI detection method based on asynchronous processing of video streams with timestamp alignment is shown in the figure. Three threads are created: a play thread, a video stream reading thread, and an AI detection thread. The read thread is responsible for decompressing frames from the camera video decoding module to obtain each image frame; the detect thread is responsible for detecting the latest frame collected by the read thread; and the play thread is responsible for displaying image frames with detection results.

[0058] Furthermore, the read thread is configured with a queue, which stores the map structure corresponding to the image frame. Each image frame collected by the read thread will be sent to the queue of the map structure. The map structure uses the timestamp time as the key, the image frame mat and the box coordinates (also known as the detection result) box as the value, and the format is: [time, mat, box]. For example, the read thread collects the first image frame and constructs the initial map structure of the first image frame; the first image frame and its timestamp are sent to the detect thread for AI detection; while the detect thread performs AI detection on the first image frame, the read thread collects the second image frame and constructs the initial map structure of the second image frame; when the detect thread completes the AI ​​detection of the first image frame, the third image frame (that is, the latest image frame) is collected and the initial map structure of the third image frame is constructed; the third image frame and its timestamp are sent to the detect thread for AI detection; and so on.

[0059] On this basis, an embodiment of the present invention provides a specific implementation of an AI detection method for asynchronous processing of video streams based on timestamp alignment, including:

[0060] In step a, the read thread receives the detection result fed back by the detect thread for the most recent image frame. For example, the timestamp time and box coordinates box fed back by the detect thread for the first image frame are expressed as [time, box].

[0061] Step b, the read thread sends the latest image frame and its timestamp association to the detect thread, including: constructing an initial map structure corresponding to the latest image frame, and sending the key and the first value of the initial map structure association to the detect thread.

[0062] For the latest image frame, the key of its initial map structure is the timestamp time of the latest image frame, the first value is the latest image frame mat, and the second value is empty, indicating that the latest image frame does not have a box coordinate box. For example, construct the initial map structure of the third image frame, the key of its initial map structure is the timestamp time of the third image frame, the first value is the third image frame mat, and the second value is empty, indicating that the third image frame does not have a box coordinate box.

[0063] The third image frame mat and its timestamp time are associated and sent to the detect thread, so that the detect thread uses the detection algorithm to detect the third image frame mat, and feeds back the timestamp time and box coordinates box of the third image frame to the read thread when the subsequent detection is completed, expressed as [time, box].

[0064] While the detect thread is performing AI detection on the latest image frame, the read thread continues to read the next adjacent image frame of the latest image frame. For example, while the detect thread is performing AI detection on the third image frame, the read thread continues to collect the fourth, fifth, and so on image frames and construct the corresponding initial map structure, and executes the following steps c to f at the same time:

[0065] In step c, the read thread aligns the previous latest image frame with the detection result based on the timestamp carried by the detection result, including: taking the timestamp carried by the detection result as the retrieval condition, retrieving the initial map structure corresponding to the previous latest image frame from the queue; using the detection result, assigning the second value of the initial map structure corresponding to the previous latest image frame to obtain the target map structure corresponding to the previous latest image frame, so as to achieve alignment of the previous latest image frame with the detection result.

[0066] For example, while the detect thread is performing AI detection on the third image frame, the read thread will use the timestamp carried by the detection result of the first image frame as the retrieval condition, retrieve the initial map structure corresponding to the first image frame from the queue, and assign the retrieval structure to the second value of the initial map structure, so as to obtain the target map structure corresponding to the first image frame, thereby aligning the first image frame and the detection result.

[0067] Step d, predicting and aligning the detection results of the image frames between the previous latest image frame and the current latest image frame, including: for each image frame between the previous latest image frame and the current latest image frame, based on the detection result of the previous adjacent image frame corresponding to the image frame, predicting the detection result of the image frame, using the predicted detection result to assign the second value of the initial map structure corresponding to the image frame, to obtain the target map structure corresponding to the image frame, so as to achieve alignment of the image frame and the predicted detection result.

[0068] For the inter-array box information stored in the queue, the Kalman filter algorithm can be used to predict the frame coordinates and state of the Nth image frame based on the Nth image frame and the N-1th image frame that has previously output the frame coordinate box, and the frame coordinate box and displacement speed of the N-1th image frame, and obtain the target map structure corresponding to the Nth image frame. For example, the second image frame has not been sent to the detect thread for AI detection, that is, the second value in its corresponding initial map structure is still empty. At this time, it is necessary to use the frame coordinate box in the target map structure corresponding to the first image frame to predict the frame coordinate box in the map structure corresponding to the second image frame.

[0069] Step e: according to the queue structure entry and exit principle, the target map structure stored in the queue is sent to the play thread. In one example, the target map structure stored in the queue can be sent to the play thread according to the queue first-in-first-out principle.

[0070] Step f, the play thread uses OpenCV to read and display the image frame and detection results in the target map structure. In one example, the play thread obtains the frame coordinates and image frame that have been aligned according to the timestamp, uses opencv to read the image frame and frame coordinates, draws and displays.

[0071] Step g: when the detect thread completes the AI ​​detection for the latest image frame, repeat steps a to f until the AI ​​detection task is stopped.

[0072] For example, when the detect thread completes the AI ​​detection of the third image frame, it feeds back the timestamp time and box coordinates box of the third image frame to the read thread, expressed as [time, box]; the read thread sends the latest fifth image frame and its timestamp association to the detect thread to continue AI detection; in the process of AI detection of the fifth image frame, the read thread will realize the alignment between the third image frame and the box coordinates, and realize the prediction of the box coordinates of the fourth image frame, and send the third image frame, the fourth image frame and their box coordinates to the play thread for display; this cycle repeats until the AI ​​detection task is completed.

[0073] In summary, the AI ​​detection method for asynchronous processing of video streams based on timestamp alignment provided by the embodiment of the present invention has at least the following characteristics:

[0074] (1) The playback thread does not need to wait for the response time of the AI ​​detection thread. During the process of the AI ​​detection thread processing the request, the playback thread can still play the detection picture in real time;

[0075] (2) It solves the technical problems of low rendering performance, significant resource occupation and delay in serial synchronous processing in traditional AI detection technology;

[0076] (III) In the practice, a solution based on timestamp alignment was proposed to solve the problem of misalignment between detection structure and image frame in asynchronous process;

[0077] (iv) Due to the low resource usage and real-time processing, drone AI detection does not rely on server-side deployment and can be deployed locally on a PC;

[0078] (V) There is a certain delay only at the beginning of the AI ​​detection task. As the task progresses, the detection image can be played back in real time to a better effect.

[0079] Based on the above embodiments, an embodiment of the present invention provides an AI detection device for asynchronous processing of video streams based on timestamp alignment. The device is applied to a terminal, and the terminal is configured with a playback thread, a video stream reading thread, and an AI detection thread. Figure 4 The structure diagram of an AI detection device for asynchronous processing of video streams based on timestamp alignment is shown, and the device includes:

[0080] The sending and receiving module 402 is used for: the video stream reading thread receives the detection result fed back by the AI ​​detection thread for the previous latest image frame, and sends the current latest image frame and its timestamp to the AI ​​detection thread;

[0081] The image frame detection module 404 is used to: the AI ​​detection thread performs AI detection on the latest image frame;

[0082] The image frame and result alignment module 406 is used to: during the period when the AI ​​detection thread performs AI detection on the current latest image frame, the video stream reading thread aligns the previous latest image frame and the detection result based on the timestamp carried by the detection result, and predicts and aligns the detection results of the image frames between the previous latest image frame and the current latest image frame, and sends the aligned image frames and detection results to the playback thread;

[0083] The image frame and result display module 408 is used to: play the thread to display the aligned image frame and the detection result;

[0084] The repeated calling module 410 is used to: when the AI ​​detection thread completes the AI ​​detection on the current latest image frame, repeat the sending and receiving module 402, the image frame detection module 404, the image frame and result alignment module 406 and the image frame and result display module 408 until the preset stop condition is met.

[0085] The AI ​​detection device for asynchronous processing of video streams based on timestamp alignment provided in an embodiment of the present invention uses a playback thread, a video stream reading thread and an AI detection thread to perform asynchronous processing on the collected image frames, so that the playback thread does not need to wait for the response time of the AI ​​detection thread. When the AI ​​detection thread performs AI detection on the current latest image frame, the playback thread can still play the image frame and detection results in real time, thereby effectively alleviating the problems of low rendering performance, significant resource occupation and delay in serial synchronous processing in traditional AI detection technology. In addition, the alignment between image frames and detection results is achieved by using timestamps, which can solve the problem of misalignment between image frames and detection results during asynchronous processing.

[0086] In one implementation, the video stream reading thread is configured with a queue, and the queue stores a map structure corresponding to the image frame; the sending and receiving module 402 is specifically used for:

[0087] Collect the latest image frame;

[0088] Construct an initial map structure corresponding to the current latest image frame; wherein the key of the initial map structure is the timestamp of the current latest image frame, the first value is the current latest image frame, and the second value is empty;

[0089] Send the key and first value association of the initial map structure to the AI ​​detection thread.

[0090] In one implementation, the image frame and result alignment module 406 is specifically configured to:

[0091] Using the timestamp carried by the detection result as the retrieval condition, retrieve the initial map structure corresponding to the previous latest image frame from the queue;

[0092] Using the detection result, the second value of the initial map structure corresponding to the previous latest image frame is assigned to obtain the target map structure corresponding to the previous latest image frame, so as to align the previous latest image frame with the detection result.

[0093] In one implementation, the image frame and result alignment module 406 is specifically configured to:

[0094] For each image frame between the previous latest image frame and the current latest image frame, the detection result of the image frame is predicted based on the detection result of the previous adjacent image frame corresponding to the image frame, and the second value of the initial map structure corresponding to the image frame is assigned using the predicted detection result to obtain the target map structure corresponding to the image frame, so as to align the image frame and the predicted detection result.

[0095] In one implementation, the image frame and result alignment module 406 is specifically configured to:

[0096] According to the structure entry and exit principle of the queue, the target map structure saved in the queue is sent to the playback thread.

[0097] In one implementation, the image frame and result display module 408 is specifically configured to:

[0098] The playback thread uses OpenCV to read and display the image frames and detection results in the target map structure.

[0099] In one embodiment, a reading module is further included, which is used to: while the AI ​​detection thread performs AI detection on the current latest image frame, the video stream reading thread continues to read the next adjacent image frame of the current latest image frame.

[0100] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0101] An embodiment of the present invention provides a terminal. Specifically, the terminal includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned implementation modes.

[0102] Figure 5 A schematic diagram of the structure of a terminal provided in an embodiment of the present invention, the terminal 100 includes: a processor 50, a memory 51, a bus 52 and a communication interface 53, wherein the processor 50, the communication interface 53 and the memory 51 are connected via the bus 52; the processor 50 is used to execute an executable module stored in the memory 51, such as a computer program.

[0103] The memory 51 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 53 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0104] The bus 52 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0105] Among them, the memory 51 is used to store programs, and the processor 50 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 50 or implemented by the processor 50.

[0106] The processor 50 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 50. The above processor 50 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 51, and the processor 50 reads the information in the memory 51 and completes the steps of the above method in combination with its hardware.

[0107] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0108] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0109] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An AI detection method for asynchronous processing of video streams based on timestamp alignment, characterized in that: The method is applied to a terminal, the terminal is configured with a playback thread, a video stream reading thread, and an AI detection thread, and the method includes: Step 1, the video stream reading thread receives the detection result fed back by the AI ​​detection thread for the previous latest image frame, and sends the current latest image frame and its timestamp association to the AI ​​detection thread; Step 2, the AI ​​detection thread performs AI detection on the latest image frame; Step 3, while the AI ​​detection thread performs AI detection on the current latest image frame, the video stream reading thread aligns the previous latest image frame and the detection result based on the timestamp carried by the detection result, and predicts and aligns the detection results of the image frames between the previous latest image frame and the current latest image frame, and sends the aligned image frames and the detection results to the playback thread; Step 4, the playback thread displays the aligned image frames and the detection results; Step 5: When the AI ​​detection thread completes the AI ​​detection on the latest image frame, repeat steps 1 to 4 until a preset stop condition is met.

2. The AI ​​detection method for asynchronous processing of video streams based on timestamp alignment according to claim 1 is characterized in that: The video stream reading thread is configured with a queue, and the queue stores a map structure corresponding to the image frame; Sending the latest image frame and its timestamp association to the AI ​​detection thread includes: Construct an initial map structure corresponding to the latest image frame; wherein the key of the initial map structure is the timestamp of the latest image frame, the first value is the latest image frame, and the second value is empty; The key of the initial map structure and the first value are associated and sent to the AI ​​detection thread.

3. The AI ​​detection method for asynchronous processing of video streams based on timestamp alignment according to claim 2 is characterized in that: Based on the timestamp carried by the detection result, aligning the previous latest image frame with the detection result, including: Using the timestamp carried by the detection result as a retrieval condition, retrieving the initial map structure corresponding to the previous latest image frame from the queue; Using the detection result, the second value of the initial map structure corresponding to the previous latest image frame is assigned to obtain the target map structure corresponding to the previous latest image frame, so as to align the previous latest image frame with the detection result.

4. The AI ​​detection method for asynchronous processing of video streams based on timestamp alignment according to claim 2 is characterized in that: Predicting and aligning the detection results of the image frames between the previous latest image frame and the current latest image frame, including: For each image frame between the previous latest image frame and the current latest image frame, the detection result of the image frame is predicted based on the detection result of the previous adjacent image frame corresponding to the image frame, and the second value of the initial map structure corresponding to the image frame is assigned using the predicted detection result to obtain the target map structure corresponding to the image frame, so as to align the image frame and the predicted detection result.

5. The AI ​​detection method for asynchronous processing of video streams based on timestamp alignment according to claim 3 or 4, characterized in that: Sending the aligned image frame and the detection result to the playback thread includes: According to the structure entry and exit principle of the queue, the target map structure saved in the queue is sent to the playback thread.

6. The AI ​​detection method for asynchronous processing of video streams based on timestamp alignment according to claim 3 or 4, characterized in that: The playback thread displays the aligned image frame and the detection result, including: The playback thread uses OpenCV to read and display the image frame and the detection result in the target map structure.

7. The AI ​​detection method for asynchronous processing of video streams based on timestamp alignment according to claim 1, characterized in that: The method further comprises: While the AI ​​detection thread is performing AI detection on the current latest image frame, the video stream reading thread continues to read the next adjacent image frame of the current latest image frame.

8. An AI detection device for asynchronous processing of video streams based on timestamp alignment, characterized in that: The device is applied to a terminal, the terminal is configured with a playback thread, a video stream reading thread and an AI detection thread, and the device includes: The sending and receiving module is used for: the video stream reading thread receives the detection result fed back by the AI ​​detection thread for the previous latest image frame, and sends the current latest image frame and its timestamp to the AI ​​detection thread; The image frame detection module is used for: the AI ​​detection thread performs AI detection on the latest image frame; An image frame and result alignment module, used for: during the period when the AI ​​detection thread performs AI detection on the current latest image frame, the video stream reading thread aligns the previous latest image frame and the detection result based on the timestamp carried by the detection result, and predicts and aligns the detection results of the image frames between the previous latest image frame and the current latest image frame, and sends the aligned image frames and the detection results to the playback thread; An image frame and result display module, used for: the playback thread displays the aligned image frame and the detection result; The repeated calling module is used to: when the AI ​​detection thread completes the AI ​​detection for the current latest image frame, repeat the sending and receiving module, the image frame detection module, the image frame and result alignment module and the image frame and result display module until a preset stop condition is met.

9. A terminal, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images