Data processing method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a nonvolatile storage medium and electronic equipment. The method comprises the following steps: reading an image frame in a video stream to be processed, and storing the image frame into an input cache queue; extracting image frames from the input buffer queue, and processing the ith image frame by using a visual identification algorithm based on artificial intelligence to obtain a processing result; superposing the processing result to the ith image frame to obtain an ith target image frame, and sending the ith target image frame to an output cache queue; and extracting the image frames from the output buffer queue, and if the processing result is superposed in the extracted jth image frame, sending the jth image frame to the to-be-processed video stream. The technical problem that the video stream and the processing result obtained by processing the video stream through a visual identification algorithm based on artificial intelligence cannot be displayed at the same time is solved.

Description

Technical Field

[0001] The present application relates to the field of computer network streaming media, and more specifically, to a data processing method and device, a non-volatile storage medium, and an electronic device. Background Art

[0002] For various audio and video live broadcasts, especially security monitoring live video streams, more and more visual AI algorithms are introduced for processing, such as personnel recognition, behavior recognition, and safety regulations. For live streams, visual AI algorithms are introduced for analysis. The relevant method is to extract video frames from the live video stream at fixed time intervals, convert them into pictures, and use them as input for the AI ​​algorithm. After algorithm processing, the results are either output as pure data or superimposed on the input video frame and then output as pictures, depending on the output data type.

[0003] There are two problems with the above method: On the one hand, because the live stream playback and AI analysis results are displayed separately, it is not intuitive and clear from the user's perspective. On the other hand, the video frame rate of the live stream is usually between 15 and 60fps, that is, the interval between adjacent frames is between 16.7 and 66.7ms. If the visual AI algorithm cannot match the processing time of a single video frame to the time interval between adjacent frames, the live stream playback picture and the AI ​​algorithm result output picture will be out of sync in display - that is, there is a significant time difference between the live stream playback picture and the AI ​​analysis result picture.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The present application provides a data processing method and device, a non-volatile storage medium, and an electronic device to at least solve the technical problem of being unable to simultaneously display a video stream and a processing result obtained by processing the video stream using an artificial intelligence-based visual recognition algorithm.

[0006] According to one aspect of the present application, a data processing method is provided, including: reading image frames in a video stream to be processed, and storing the image frames in an input cache queue; extracting image frames from the input cache queue, and processing the i-th image frame using an artificial intelligence-based visual recognition algorithm to obtain a processing result, wherein i is a positive integer; superimposing the processing result on the i-th image frame to obtain an i-th target image frame, and sending the i-th target image frame to an output cache queue; extracting image frames from the output cache queue, and if the extracted j-th image frame has the processing result superimposed on it, sending the j-th image frame to the video stream to be processed, wherein j is a positive integer greater than 1.

[0007] Optionally, the method also includes: if the extracted j-th image frame does not have a superimposed processing result, determining whether the j-th image frame is a lens transition frame; if the j-th image frame is a lens transition frame, sending the j-th image frame to the video stream to be processed.

[0008] Optionally, the method also includes: when the jth image frame is not a lens transition frame, obtaining the processing result of the j-1th image frame, and superimposing the processing result of the j-1th image frame on the jth image frame to obtain the jth target image frame; sending the jth target image frame to the video stream to be processed.

[0009] Optionally, determining whether the j-th image frame is a lens transition frame includes: determining a first histogram of a brightness component of the j-th image frame, and determining a second histogram of a brightness component of the j-1-th image frame; normalizing the first histogram and the second histogram to obtain first histogram data and second histogram data; determining a Bhattacharyya distance between the j-th image frame and the j-1-th image frame according to a first mean of the first histogram data, a first number of continuous intervals in the first histogram data, a second mean of the second histogram data, and a second number of continuous intervals in the second histogram data; if the Bhattacharyya distance is greater than a preset threshold, determining that the j-th image frame is a lens transition frame; if the Bhattacharyya distance is not greater than the preset threshold, determining that the j-th image frame is not a lens transition frame.

[0010] Optionally, the method also includes: determining whether a first difference between the moment indicated in the timestamp of the i-th image frame extracted and the target moment is less than a second difference between the moment indicated in the timestamp of other image frames and the target moment; if the first difference between the moment indicated in the timestamp of the i-th image frame extracted and the target moment is less than the second difference between the moment indicated in the timestamp of other image frames and the target moment, processing the i-th image frame using an artificial intelligence-based visual recognition algorithm; if the first difference between the moment indicated in the timestamp of the i-th image frame extracted and the target moment is not less than the second difference between the moment indicated in the timestamp of other image frames and the target moment, sending the i-th image frame to the output cache queue.

[0011] Optionally, after storing the image frames in the input cache queue, the method further includes: adjusting the length of the input cache queue according to the frame rate of the video stream to be processed, the time interval between adjacent image frames in the video stream to be processed, and the time consumed by the target algorithm to process the image frames.

[0012] Optionally, processing the i-th image frame includes: parsing video content in the video stream to be processed; selecting a target algorithm that matches the video content; and processing the i-th image frame using the target algorithm.

[0013] According to another aspect of the present application, a data processing method is also provided, including: reading image frames from a video stream to be processed and caching the image frames; processing the cached image frames using an artificial intelligence-based visual recognition algorithm to obtain processing results; superimposing the processing results on the image frames to obtain target image frames; adding the target image frames to the video stream to be processed, and outputting the video stream to be processed.

[0014] According to another aspect of the present application, a data processing device is also provided, including: a storage module, used to read image frames in a video stream to be processed, and store the image frames in an input cache queue; a processing module, used to extract image frames from the input cache queue, and process the i-th image frame using an artificial intelligence-based visual recognition algorithm to obtain a processing result, wherein i is a positive integer; a first sending module, used to superimpose the processing result on the i-th image frame to obtain the i-th target image frame, and send the i-th target image frame to an output cache queue; a second sending module, used to extract image frames from the output cache queue, and if the extracted j-th image frame has the processing result superimposed on it, send the j-th image frame to the video stream to be processed, wherein j is a positive integer greater than 1.

[0015] According to another aspect of the present application, a non-volatile storage medium is provided, the storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above data processing method.

[0016] According to another aspect of the present application, an electronic device is provided, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the above data processing method is executed when the program is run.

[0017] According to yet another aspect of the present application, a computer program is provided, wherein the above data processing method is implemented when the computer program is executed by a processor.

[0018] According to another aspect of the present application, a computer program product is provided, which includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above data processing method is implemented.

[0019] In the present application, the image frames are read from the video stream to be processed and stored in the input cache queue; the image frames are extracted from the input cache queue, and the i-th image frame is processed by a visual recognition algorithm based on artificial intelligence to obtain a processing result, wherein i is a positive integer; the processing result is superimposed on the i-th image frame to obtain the i-th target image frame, and the i-th target image frame is sent to the output cache queue; the image frames are extracted from the output cache queue, and if the j-th image frame extracted has the processing result superimposed on it, the j-th image frame is sent to the video stream to be processed, wherein j is a positive integer greater than 1. The purpose of simultaneously displaying the video stream and the processing result obtained by processing the video stream by the visual recognition algorithm based on artificial intelligence is achieved, thereby realizing the technical effect of allowing users to intuitively view the processing result of the artificial intelligence algorithm from the video stream, thereby solving the technical problem that the video stream and the processing result obtained by processing the video stream by the visual recognition algorithm based on artificial intelligence cannot be displayed simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a flow chart of a data processing method according to an embodiment of the present application;

[0022] Figure 2 is a flow chart of another data processing method according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a data processing method according to an embodiment of the present application;

[0024] Figure 4 is a structural diagram of a data processing device according to an embodiment of the present application;

[0025] Figure 5 It is a hardware structure block diagram of a computer terminal according to a data processing method of an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] According to an embodiment of the present application, a method embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0030] Step S102: reading image frames from the video stream to be processed, and storing the image frames in an input buffer queue.

[0031] Step S104, extracting image frames from the input cache queue, and processing the i-th image frame using a visual recognition algorithm based on artificial intelligence to obtain a processing result, where i is a positive integer.

[0032] The processing thread reads an image frame from the input cache queue for processing. Due to the dynamic adjustment of the cache queue, the processing thread may need to wait until enough frames accumulate in the cache queue to meet the processing conditions. The i-th image frame (where i is a positive integer, indicating the order of the image frames in the queue) is processed using a preset visual AI algorithm. AI algorithms include but are not limited to face recognition, target detection, behavior analysis, etc. The algorithm processing time will affect the calculation of the length of the dynamic cache queue. After processing by the AI ​​algorithm, a series of data results will be obtained, such as the coordinates and category of the identified target.

[0033] Step S106 , superimposing the processing result on the ith image frame to obtain the ith target image frame, and sending the ith target image frame to the output buffer queue.

[0034] The AI ​​algorithm processing results obtained in step S104, such as the target detection frame, classification label and other visual information, are superimposed on the i-th image frame to form the i-th target image frame. The i-th target image frame processed and superimposed with the AI ​​results is stored in the output cache queue. The output cache queue also needs to be initialized and dynamically adjusted according to the processing rate of the output thread and the user viewing experience.

[0035] Step S108, extracting image frames from the output buffer queue, and if the j-th image frame extracted has the processing result superimposed on it, sending the j-th image frame to the video stream to be processed, where j is a positive integer greater than 1.

[0036] The output thread extracts image frames from the output cache queue. This can be done based on the display timestamp (PTS) to ensure that the output of the image frame is synchronized with the original live stream. Determine whether the jth image frame extracted (j is a positive integer greater than 1) has been superimposed with the processing results of the visual AI algorithm. If the image frame already contains the processing results, it means that it has been processed by the processing thread and stored in the output cache queue.

[0037] If the processing result has been superimposed on the j-th image frame, the output thread sends the target image frame to the video stream to be processed or directly outputs it to the end user to achieve real-time display of the visual AI algorithm processing results.

[0038] According to the above steps, by effectively processing each frame in the video stream, the real-time overlay display of the AI ​​algorithm processing results is ensured. At the same time, through dynamic cache queue management and lens transition frame judgment, the processing efficiency and user experience are improved.

[0039] The following Figure 1 The steps shown are exemplary and explanatory.

[0040] According to some optional embodiments of the present application, if the extracted j-th image frame does not have a superimposed processing result, it is determined whether the j-th image frame is a lens transition frame; if the j-th image frame is a lens transition frame, the j-th image frame is sent to the video stream to be processed.

[0041] The output thread extracts the jth image frame from the output cache queue, indicating the order of the frame in the output cache queue. For the jth image frame, the lens transition value with the previous frame (i.e., the j-1th image frame) is calculated to determine whether the visual difference between the two frames meets the lens transition standard. It can be calculated based on the Y component histogram comparison using the Bhattacharyya distance. The calculated lens transition value is compared with the preset lens transition threshold. If the lens transition value exceeds the threshold, it means that the jth image frame has a significant visual difference compared to the previous frame, which may be the beginning of a new lens or scene. When the jth image frame is judged to be a lens transition frame, the jth image frame is sent to the video stream to be processed or the output stream to ensure that the user can see the new picture in time when the video content changes significantly, rather than being overwritten by outdated AI processing results.

[0042] Through the above steps, the lens changes in the video stream are dynamically identified, and the superposition and output of the AI ​​processing results are adjusted accordingly. In the lens transition frames (i.e. frames with significant visual differences), since the current frame may not have been processed by the AI ​​algorithm (perhaps because the processing takes a long time and the current frame has not been processed in time), the system chooses not to superimpose any AI data, but directly outputs it to prioritize the continuity and real-time nature of the video content. For non-lens transition frames, if the previous frame has been processed by the AI ​​algorithm, the processing result of the previous frame is superimposed on the current frame and output to compensate for the delay of AI processing, so that users will not feel obvious interruptions when watching.

[0043] In some optional embodiments of the present application, after the image frames are stored in the input cache queue, the following steps may also be performed: adjust the length of the input cache queue according to the frame rate of the video stream to be processed, the time interval between adjacent image frames in the video stream to be processed, and the time taken by the target algorithm to process the image frames.

[0044] Initialize the input buffer queue to temporarily store the image frames in the live stream. The length of the input buffer queue is dynamically calculated based on the following formula and the time consumption of the AI ​​algorithm to ensure that it can accommodate a sufficient number of frames to meet the needs of algorithm processing.

[0045] The input thread continuously reads new image frames from the live video stream. This can be supported by the video stream decoder, which decodes the compressed video data into image frames and provides metadata such as decoding timestamp (DTS) and presentation timestamp (PTS). The read image frames and their metadata are stored in the input buffer queue. To avoid data overflow, it is necessary to wait when the queue is full or use a circular queue mechanism so that old frames are overwritten by new frames.

[0046]

[0047] Where a is the dynamic cache queue length of the input video stream (unit: frame), χ is the frame rate of the input video stream (unit: fps), γ is the time interval between adjacent frames of the input video stream (unit: ms), and β is the time taken by the AI ​​algorithm to process one frame (unit: ms).

[0048] It can be understood that according to formula (1) and the input thread and processing thread workflow in the embodiment, the minimum and necessary input frame buffering is performed on the input video frame sequence by dynamically adjusting the buffer queue size. Compared with the traditional fixed buffer input frame queue size or the empirically set buffer frame queue size, the effect is significantly improved.

[0049] According to other optional embodiments of the present application, when the jth image frame is not a lens transition frame, the processing result of the j-1th image frame is obtained, and the processing result of the j-1th image frame is superimposed on the jth image frame to obtain the jth target image frame; the jth target image frame is sent to the video stream to be processed.

[0050] After determining that the j-th image frame is not a shot transition frame, the output thread will further process the frame. This means that the current frame is visually continuous with the previous frame (i.e., the j-1th image frame) and there is no significant scene change. The output thread retrieves the processing results of the j-1th image frame from the output cache queue or related storage, which have been superimposed on the image frame and stored after the previous frame was processed by the AI ​​algorithm. Confirm whether the processing results of the j-1th image frame are still valid, that is, whether superimposing the results on the current frame can accurately reflect the information of the current frame. Superimpose the processing results of the j-1th image frame (such as target detection box, classification label, etc.) on the j-1th image frame to form the j-th target image frame. Send the j-th target image frame processed and superimposed with the AI ​​algorithm results to the video stream to be processed or directly output to the user to ensure that the user can see a continuous video stream with AI processing results even in the case of AI algorithm processing delay.

[0051] In some optional embodiments of the present application, determining whether the j-th image frame is a shot transition frame can be achieved by the following method: determining a first histogram of the brightness component of the j-th image frame, and determining a second histogram of the brightness component of the j-1-th image frame; normalizing the first histogram and the second histogram to obtain first histogram data and second histogram data; determining a Bhattacharyya distance between the j-th image frame and the j-1-th image frame according to a first mean of the first histogram data, a first number of continuous intervals in the first histogram data, a second mean of the second histogram data, and a second number of continuous intervals in the second histogram data; when the Bhattacharyya distance is greater than a preset threshold, determining that the j-th image frame is a shot transition frame; when the Bhattacharyya distance is not greater than the preset threshold, determining that the j-th image frame is not a shot transition frame.

[0052] Extract the brightness component (usually the Y component, corresponding to the brightness information in the YUV color space) from the jth image frame. Count the distribution of pixel values ​​in the brightness component of the jth image frame to obtain the first histogram. This histogram describes the frequency of occurrence of pixels with different brightness values ​​in the image. Similarly, extract the brightness component from the j-1th image frame. Count the distribution of pixel values ​​in the brightness component of the j-1th image frame to obtain the second histogram, which describes the distribution of brightness values ​​in the previous frame.

[0053] The first histogram data is converted to a standardized value range, such as [0,255], to eliminate differences caused by factors such as image size and resolution. The second histogram data is normalized in the same way to ensure that the two histogram data can be compared at the same scale.

[0054] Specifically, the normalization process can be reflected by the following formula:

[0055]

[0056] The value range of the pixel point is [m, n], and the histogram of the Y component is normalized and converted to a value range of [i, j] (i = 0, j = 255).

[0057] Based on the normalized first histogram data, the mean of its pixel values ​​is calculated. The number of pixel values ​​in the continuous interval in the first histogram is counted. Based on the normalized second histogram data, the mean of its pixel values ​​is calculated. The number of pixel values ​​in the continuous interval in the second histogram is counted, which is also used to analyze the local brightness changes of the image.

[0058] According to the first mean, the first quantity, the second mean, and the second quantity, the visual similarity between the j-th image frame and the j-1-th image frame is calculated using the Bhattacharyya distance formula, wherein the Bhattacharyya distance is a measure of the difference in probability distribution, and is used here to evaluate the difference between the brightness histograms of the two frames of images.

[0059] Specifically, the algorithm for calculating the similarity between two frames using Bhattacharyya distance is as follows:

[0060]

[0061] in, is the mean of the histogram data, and N is the number of BINs in the histogram. In the histogram, "the number of BINs" refers to the number of small intervals or categories used to divide the data range. Each "BIN" represents a small range of values ​​or categories, and the histogram reflects the distribution of data by showing the frequency or frequency in these "BINs".

[0062] The calculated Bhattacharyya distance is compared with the preset shot transition threshold. If the Bhattacharyya distance is greater than the preset threshold, it means that there is a significant visual difference between the j-th image frame and the j-1-th image frame, and the j-th image frame can be judged as a shot transition frame. If the Bhattacharyya distance is not greater than the preset threshold, it means that the two frames are visually continuous and have not undergone significant scene changes. Therefore, it can be judged that the j-th image frame is not a shot transition frame and it still belongs to the current shot or scene.

[0063] It can be understood that according to formula (3) and the processing thread workflow in the embodiment, the frames that have not undergone AI calculation are also superimposed with the AI ​​algorithm processing results. At the same time, if it is a shot transition frame, the AI ​​algorithm result of the previous frame is not copied and superimposed on the current frame, ensuring that the AI ​​algorithm result is not excessively diffused, resulting in a phenomenon where the AI ​​algorithm result does not match the video content of the current frame.

[0064] As some optional embodiments of the present application, it is determined whether a first difference between the moment indicated in the timestamp of the i-th image frame extracted and the target moment is less than a second difference between the moment indicated in the timestamp of other image frames and the target moment; if the first difference between the moment indicated in the timestamp of the i-th image frame extracted and the target moment is less than the second difference between the moment indicated in the timestamp of other image frames and the target moment, the i-th image frame is processed using an artificial intelligence-based visual recognition algorithm; if the first difference between the moment indicated in the timestamp of the i-th image frame extracted and the target moment is not less than the second difference between the moment indicated in the timestamp of other image frames and the target moment, the i-th image frame is sent to the output cache queue.

[0065] The processing thread sets a target time, which can be based on the system real-time time or the expected playback time of the video stream, to determine how close the image frame is to the real-time display. Get the timestamp of the i-th image frame, calculate the difference between its indicated time and the target time, and record it as the first difference. For other image frames in the input cache queue, also calculate the difference between their timestamp indicated time and the target time, and record it as the second difference. The processing thread needs to traverse the timestamps of all frames in the queue and perform difference calculations. Determine whether the first difference is the smallest: compare the first difference of the i-th image frame with the second difference of other image frames to determine whether the i-th image frame is the image frame closest to the target time. If the first difference is less than the second difference of all other frames, it means that the i-th image frame is the frame closest to the target time (the latest frame), and the i-th image frame is preferentially processed using the artificial intelligence-based visual recognition algorithm. This is because processing the frame closest to the target time can ensure that the AI ​​algorithm processing results are as synchronized as possible with the real-time video playback, providing users with the best viewing experience. If the first difference between the time indicated by the timestamp of the i-th image frame and the target time is not less than the second difference of other frames, it means that the current frame is not the frame closest to the target time. At this time, the processing thread will not immediately perform AI algorithm processing on the current frame, but will send it directly to the output cache queue to wait for subsequent processing or output. For frames that are not immediately processed by the AI ​​algorithm, the output thread will decide whether to output directly or superimpose the AI ​​algorithm processing results before outputting them, based on whether they are lens transition frames and the processing results of the previous frame. This dynamic strategy ensures that the smoothness and continuity of video playback can be guaranteed even in the case of AI algorithm processing delays.

[0066] Optionally, processing the i-th image frame may be achieved by the following method: parsing video content in the video stream to be processed; selecting a target algorithm that matches the video content; and processing the i-th image frame using the target algorithm.

[0067] Before parsing the video stream to be processed, the system first identifies the format of the video stream, including encoding method (such as H.264, H.265, etc.), frame rate, resolution and other parameters to ensure the compatibility and efficiency of the subsequent processing process. The input thread continuously reads video frames from the video stream to be processed and stores each frame in the input dynamic cache queue. When reading, the decoding timestamp (DTS) and presentation timestamp (PTS) of each frame are obtained at the same time for time synchronization and optimization in the subsequent processing flow. The image frames in the input cache queue are preliminarily parsed to analyze their content features, such as the type of objects in the frame, color distribution, texture information, etc.

[0068] A visual AI algorithm library is built-in or external, which contains a variety of algorithms, such as object recognition, behavior analysis, scene detection, etc. Each algorithm is optimized for different types or characteristics of video content. Based on the preliminary content analysis results of the i-th image frame, the type and requirements of the video content are judged. For example, if the frame contains a large number of pedestrians, human recognition and behavior analysis algorithms may be selected; if the frame contains complex scenes, scene detection or environmental analysis algorithms are selected. According to the type and requirements of the video content, the most appropriate AI algorithm is selected from the algorithm library as the target algorithm. When selecting, the processing efficiency and resource consumption of the algorithm must also be considered to ensure that it can process the video stream in real time or near real time.

[0069] The processing thread reads the i-th image frame from the input dynamic cache queue and converts the image frame into the input format required by the AI ​​algorithm, such as an RGB image or a specific image preprocessing step. The i-th image frame is deeply analyzed using the selected target algorithm, which may include processes such as object detection, feature extraction, and classification recognition, depending on the type of algorithm selected. After the method processing is completed, the obtained analysis results (such as object bounding boxes, classification labels, key points, etc.) are superimposed on the i-th image frame to form a target image frame containing the AI ​​processing results. The processed target image frame is output to the output cache queue, waiting for the output thread to determine the final output order and method based on the lens over-judgment and time synchronization strategy.

[0070] Through the above steps, the function of dynamically selecting and applying AI algorithms according to the content of the video stream is realized, which improves the pertinence and efficiency of AI analysis. At the same time, by independently analyzing and processing each frame, it ensures that the results of the AI ​​algorithm can be accurately and intuitively presented on the live streaming video, enhancing the interactivity and information acquisition ability of users when watching videos. This implementation method is particularly suitable for live video processing scenarios with rich and diverse content and high real-time requirements, such as security monitoring, sports live broadcasts, online education, etc.

[0071] The embodiment of the present application also provides a data processing method, including: reading image frames from a video stream to be processed and caching the image frames; processing the cached image frames using an artificial intelligence-based visual recognition algorithm to obtain processing results; superimposing the processing results on the image frames to obtain target image frames; adding the target image frames to the video stream to be processed, and outputting the video stream to be processed.

[0072] When the system starts, an input dynamic cache queue is created to store image frames read from the video stream. The input thread reads image frames from the video stream to be processed and checks whether the video stream has ended. If the video stream has not ended, the read image frames are stored in the input dynamic cache queue and the next frame is read. According to the frame rate of the video stream, the time interval between adjacent frames, and the time it takes for the AI ​​algorithm to process a frame of image, the length of the input dynamic cache queue is dynamically adjusted to ensure that the number of cached image frames can match the algorithm processing capability and video stream characteristics.

[0073] The processing thread selects an image frame from the input dynamic cache queue and determines whether the current frame needs to be processed by the AI ​​algorithm. This step ensures that the image frame processed by the AI ​​algorithm is the most relevant and avoids unnecessary waste of computing resources. If the current frame needs to be processed by the AI ​​algorithm, the image frame is deeply analyzed using the artificial intelligence-based visual recognition algorithm. After the AI ​​algorithm is processed, a processing result containing recognition results, labels, bounding boxes and other information is generated.

[0074] The information in the processing result is superimposed on the image frame to form the target image frame. This includes drawing a bounding box in the image, adding classification labels, displaying key points, etc., to intuitively display the analysis results of the AI ​​algorithm. Ensure that the format of the target image frame is compatible with the requirements of the output stream to prepare for subsequent output. The processed target image frame is added to the output cache queue, waiting for the output thread to perform the final output operation. The output thread extracts the target image frame from the output cache queue and calculates whether the lens transition value between the current frame and the previous frame exceeds the preset lens transition threshold. If the lens transition threshold is exceeded, it means that there is a significant visual difference between the current frame and the previous frame. The output thread directly outputs the target image frame to the output stream without reusing the AI ​​processing result of the previous frame. If the lens transition threshold is not exceeded, it means that the current frame is visually continuous with the previous frame. The output thread copies and superimposes the AI ​​processing result of the previous frame to the current frame, and then outputs it to the output stream. Finally, the output thread outputs the processed image frame in the form of a video stream based on the above judgment and strategy, and the user can intuitively see the analysis results of the visual AI algorithm in the live stream.

[0075] It should be noted that the preferred implementation of the above embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0076] Figure 2 is a flowchart of another data processing method according to an embodiment of the present application, Figure 3 is a schematic diagram of a data processing method according to an embodiment of the present application, such as Figure 2 and Figure 3 As shown, the method comprises the following steps:

[0077] Step S201, initializing and reading the cache: When the system starts, the input dynamic cache queue is initialized to ensure that the cache queue is empty and ready to receive new video frames.

[0078] Step S202, loop reading and caching video frames. The input thread starts loop reading video frames in the input stream and checks whether the input stream has ended. If the input stream has not ended, the next step is executed; if the input stream has ended, the input thread workflow terminates.

[0079] Step S203, cache video frames. Store the read video frames into the input dynamic cache queue. After storage is completed, continue to read the next frame of video until the input stream ends.

[0080] Step S204, reading a frame from the cache queue. The frame processing thread in the read cache queue reads a frame of video data from the input dynamic cache queue for analysis.

[0081] Step S205, determine whether to perform AI algorithm processing, specifically, using the following formula to determine:

[0082]

[0083] Wherein, T represents the display timestamp of the current frame (unit: ms).

[0084] Formula (4) is used to calculate whether the current frame needs to be processed by the AI ​​algorithm, based on the time difference between the time required for the AI ​​algorithm to process a frame and the real-time characteristics of the video stream.

[0085] Step S206: If the current frame needs to be processed by AI algorithm, execute the visual recognition algorithm based on artificial intelligence to identify and analyze the content in the frame. After the algorithm processing is completed, the recognition result is superimposed on the current frame in a visual way to form a target image frame.

[0086] Step S207: If AI algorithm processing is not required, the frame is directly output to the output buffer queue.

[0087] Step S208: extracting frames from the output buffer queue.

[0088] Step S209, determine whether the current frame has been processed by the AI ​​algorithm and the processing results have been superimposed. If yes, directly enter the output stage; if not, execute step S210.

[0089] Step S210, calculating and determining the shot transition value. Calculate the shot transition value between the current frame and the previous frame, that is, the degree of visual difference, and determine whether the value exceeds a preset shot transition threshold.

[0090] Step S211, if the shot transition value exceeds the threshold, the current frame is determined to be the beginning of a new shot and is directly output to the output stream without copying the processing result of the previous frame.

[0091] Step S212: If the shot transition value does not exceed the threshold, the processing result of the previous frame is copied and superimposed on the current frame, and then output to the output stream.

[0092] According to the above steps, this embodiment dynamically caches the video frames of the input stream according to the time consumption of the AI ​​algorithm, analyzes each frame to see if it is a transition frame of the previous frame, and then superimposes the processing results of the visual AI algorithm on the current frame for output. The results of the AI ​​algorithm processing are superimposed on the video output stream. Compared with the traditional presentation of the AI ​​algorithm processing results of the video stream as separate data or separate pictures, there is a significant improvement, and users have an intuitive experience of the results of the AI ​​algorithm processing.

[0093] Figure 4 is a structural diagram of a data processing device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:

[0094] The storage module 42 is used to read image frames in the video stream to be processed and store the image frames in the input buffer queue.

[0095] The processing module 44 is used to extract image frames from the input buffer queue, and process the i-th image frame using a visual recognition algorithm based on artificial intelligence to obtain a processing result, where i is a positive integer.

[0096] The first sending module 46 is used to superimpose the processing result on the i-th image frame to obtain the i-th target image frame, and send the i-th target image frame to the output buffer queue.

[0097] The second sending module 48 is used to extract image frames from the output buffer queue, and if the j-th image frame extracted has the processing result superimposed on it, send the j-th image frame to the video stream to be processed, where j is a positive integer greater than 1.

[0098] Optionally, the data processing device is also used to perform the following steps: if the extracted j-th image frame does not have the processing result superimposed on it, determine whether the j-th image frame is a shot transition frame; if the j-th image frame is a shot transition frame, send the j-th image frame to the video stream to be processed.

[0099] Optionally, the data processing device is also used to perform the following steps: when the jth image frame is not a lens transition frame, obtain the processing result of the j-1th image frame, and superimpose the processing result of the j-1th image frame on the jth image frame to obtain the jth target image frame; send the jth target image frame to the video stream to be processed.

[0100] Optionally, the data processing device is also used to perform the following steps: determine a first histogram of the brightness component of the j-th image frame, and determine a second histogram of the brightness component of the j-1-th image frame; normalize the first histogram and the second histogram to obtain first histogram data and second histogram data; determine the Bhattacharyya distance between the j-th image frame and the j-1-th image frame based on a first mean of the first histogram data, a first number of continuous intervals in the first histogram data, a second mean of the second histogram data, and a second number of continuous intervals in the second histogram data; when the Bhattacharyya distance is greater than a preset threshold, determine that the j-th image frame is a shot transition frame; when the Bhattacharyya distance is not greater than the preset threshold, determine that the j-th image frame is not a shot transition frame.

[0101] Optionally, the data processing device is also used to perform the following steps: determine whether a first difference between the moment indicated in the timestamp of the i-th image frame extracted and the target moment is less than a second difference between the moment indicated in the timestamp of other image frames and the target moment; if the first difference between the moment indicated in the timestamp of the i-th image frame extracted and the target moment is less than the second difference between the moment indicated in the timestamp of other image frames and the target moment, process the i-th image frame using an artificial intelligence-based visual recognition algorithm; if the first difference between the moment indicated in the timestamp of the i-th image frame extracted and the target moment is not less than the second difference between the moment indicated in the timestamp of other image frames and the target moment, send the i-th image frame to the output cache queue.

[0102] Optionally, the data processing device is also used to perform the following steps after storing the image frame in the input cache queue: adjust the length of the input cache queue according to the frame rate of the video stream to be processed, the time interval between adjacent image frames in the video stream to be processed, and the time consumed by the target algorithm to process the image frame.

[0103] Optionally, the processing module 44 is further configured to perform the following steps: parsing video content in the video stream to be processed; selecting a target algorithm that matches the video content; and processing the i-th image frame using the target algorithm.

[0104] It should be noted that the above Figure 4 The modules in the embodiment may be program modules (e.g., a set of program instructions for implementing a specific function) or hardware modules. For the latter, they may be expressed in the following forms, but are not limited thereto: the expression form of the above modules is a processor, or the functions of the above modules are implemented by a processor.

[0105] It should be noted that Figure 4 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0106] Figure 5 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a data processing method. Figure 5 As shown, the computer terminal 50 may include one or more (502a, 502b, ..., 502n are used to illustrate) processors 502 (the processor 502 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 504 for storing data, and a transmission module 506 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components as shown, or with Figure 5 Different configurations shown.

[0107] It should be noted that the one or more processors 502 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 50. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0108] The memory 504 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present application. The processor 502 executes various functional applications and data processing by running the software programs and modules stored in the memory 504, that is, realizing the above-mentioned data processing method. The memory 504 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 504 may further include a memory remotely arranged relative to the processor 502, and these remote memories may be connected to the computer terminal 50 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0109] The transmission module 506 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 50. In one example, the transmission module 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 506 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0110] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 50 .

[0111] It should be noted that, in some optional embodiments, the above Figure 5 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 5 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0112] It should be noted that Figure 5 The computer terminal shown is used to execute Figure 1 The data processing method shown, therefore the relevant explanations in the execution method of the above command are also applicable to the electronic device and will not be repeated here.

[0113] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above data processing method.

[0114] A program for a non-volatile storage medium to perform the following functions: read image frames in a video stream to be processed and store the image frames in an input cache queue; extract image frames from the input cache queue, process the i-th image frame using an artificial intelligence-based visual recognition algorithm to obtain a processing result, wherein i is a positive integer; superimpose the processing result on the i-th image frame to obtain the i-th target image frame, and send the i-th target image frame to an output cache queue; extract image frames from the output cache queue, and if the j-th image frame extracted has the processing result superimposed on it, send the j-th image frame to the video stream to be processed, wherein j is a positive integer greater than 1.

[0115] An embodiment of the present application further provides an electronic device, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the above data processing method is executed when the program is run.

[0116] The processor is used to run a program that performs the following functions: read image frames in a video stream to be processed and store the image frames in an input cache queue; extract image frames from the input cache queue, and process the i-th image frame using an artificial intelligence-based visual recognition algorithm to obtain a processing result, where i is a positive integer; superimpose the processing result on the i-th image frame to obtain the i-th target image frame, and send the i-th target image frame to an output cache queue; extract image frames from the output cache queue, and if the j-th image frame extracted has the processing result superimposed on it, send the j-th image frame to the video stream to be processed, where j is a positive integer greater than 1.

[0117] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0118] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0119] In the above-mentioned embodiments of the present application, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary protection measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0121] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0122] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0124] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Reading image frames from a video stream to be processed, and storing the image frames in an input buffer queue; Extracting an image frame from the input buffer queue, and processing the i-th image frame using a visual recognition algorithm based on artificial intelligence to obtain a processing result, wherein i is a positive integer; Superimposing the processing result onto the i-th image frame to obtain the i-th target image frame, and sending the i-th target image frame to the output buffer queue; Extract an image frame from the output cache queue, and if the j-th image frame extracted has the processing result superimposed on it, send the j-th image frame to the video stream to be processed, where j is a positive integer greater than 1.

2. The method according to claim 1, characterized in that The method further comprises: If the processing result is not superimposed on the j-th image frame extracted, determining whether the j-th image frame is a shot transition frame; In the case where the j-th image frame is the shot transition frame, the j-th image frame is sent to the video stream to be processed.

3. The method according to claim 2, characterized in that The method further comprises: When the j-th image frame is not the lens transition frame, obtaining the processing result of the j-1-th image frame, and superimposing the processing result of the j-1-th image frame onto the j-th image frame to obtain the j-th target image frame; The j-th target image frame is sent to the video stream to be processed.

4. The method according to claim 2, characterized in that: Determining whether the j-th image frame is a shot transition frame includes: Determining a first histogram of the brightness component of the j-th image frame, and determining a second histogram of the brightness component of the j-1-th image frame; Normalizing the first histogram and the second histogram to obtain first histogram data and second histogram data; Determine a Bhattacharyya distance between the j-th image frame and the j-1-th image frame according to a first mean of the first histogram data, a first number of continuous intervals in the first histogram data, a second mean of the second histogram data, and a second number of continuous intervals in the second histogram data; When the Bhattacharyya distance is greater than a preset threshold, the j-th image frame is determined to be the shot transition frame; when the Bhattacharyya distance is not greater than the preset threshold, the j-th image frame is determined not to be the shot transition frame.

5. The method according to claim 1, characterized in that The method further comprises: Determine whether a first difference between the time indicated in the timestamp of the extracted i-th image frame and the target time is less than a second difference between the time indicated in the timestamps of other image frames and the target time; If a first difference between the time indicated in the timestamp of the extracted i-th image frame and the target time is less than a second difference between the time indicated in the timestamps of other image frames and the target time, the i-th image frame is processed using a visual recognition algorithm based on artificial intelligence; If a first difference between the time indicated in the timestamp of the extracted i-th image frame and the target time is not less than a second difference between the time indicated in the timestamps of other image frames and the target time, the i-th image frame is sent to the output buffer queue.

6. The method according to claim 1, characterized in that After storing the image frame in the input buffer queue, the method further includes: The length of the input buffer queue is adjusted according to the frame rate of the video stream to be processed, the time interval between adjacent image frames in the video stream to be processed, and the time consumed by the target algorithm to process the image frames.

7. The method according to claim 1, characterized in that Processing the i-th image frame includes: Parsing the video content in the video stream to be processed; Selecting a target algorithm that matches the video content; The i-th image frame is processed using the target algorithm.

8. A data processing method, characterized in that: include: Reading image frames from the video stream to be processed, and caching the image frames; Using an artificial intelligence-based visual recognition algorithm to process the cached image frames, obtaining processing results; Superimposing the processing result onto the image frame to obtain a target image frame; The target image frame is added to the video stream to be processed, and the video stream to be processed is output.

9. A data processing device, characterized in that: include: A storage module, used for reading image frames in the video stream to be processed, and storing the image frames in an input buffer queue; A processing module, used for extracting image frames from the input buffer queue, and processing the i-th image frame using a visual recognition algorithm based on artificial intelligence to obtain a processing result, wherein i is a positive integer; A first sending module, used for superimposing the processing result on the i-th image frame to obtain the i-th target image frame, and sending the i-th target image frame to the output buffer queue; The second sending module is used to extract image frames from the output cache queue, and if the extracted j-th image frame has been superimposed with the processing result, send the j-th image frame to the video stream to be processed, wherein j is a positive integer greater than 1.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 8.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the data processing method according to any one of claims 1 to 8 when running.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.

Images