Image processing method and related equipment

Abstract

The embodiment of the invention provides an image processing method and related equipment, which are used for displaying a current picture and a corresponding algorithm result in real time. The method comprises the steps of obtaining to-be-processed image data collected by camera equipment; performing format conversion on the first frame image data and the second frame image data to obtain first frame display data corresponding to the first frame image data and second frame display data corresponding to the second frame image data, sequentially drawing the first frame display data and the second frame display data on a first frame display layer and a second frame display layer of the target display canvas; and when the first frame display data of the first frame display layer is analyzed based on the picture algorithm and the second frame display data is pushed to the user, drawing algorithm result data corresponding to the first frame display data on the result display layer so that the user can watch the first frame display data and the algorithm result data on the display screen.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of screen display, and in particular, to an image processing method and related devices. Background Art

[0002] The imaging devices in people's work and life have changed from traditional single intelligent devices to the coexistence of multiple device functions. However, usually, the camera screen is captured at a frequency of 30 frames per second, that is, one frame is captured approximately every 33.3 milliseconds. Since the picture algorithm processing component takes a relatively long time, greater than 33 milliseconds, if not processed, when the picture rendering result is shown to the user, the picture algorithm is still processing the frame data. Without getting the algorithm data, it will cause the user to see the frame of the picture without seeing the corresponding information of this frame, such as the number of danger signals, positions, high-risk picture warnings, etc., seriously affecting the processing accuracy of the alarm system.

[0003] The general solution to this problem is to simultaneously display the current screen and the warning information N seconds ago to the user. This solution will cause the warning information not to correspond to the screen one by one, and in serious cases, it will affect the misjudgment of the staff and cause dangerous accidents. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method and related devices for real-time displaying the current screen and the corresponding algorithm results, so that the algorithm results correspond to the current screen.

[0005] In a first aspect of the embodiments of the present application, an image processing method is provided, which is applied to an image processing device. The image processing device is connected to an imaging device. The method includes:

[0006] Obtain the to-be-processed image data collected by the imaging device; wherein, the to-be-processed image data at least includes first-frame image data and second-frame image data, and the second-frame image data is the next-frame image data of the first-frame image data;

[0007] Convert the formats of the first-frame image data and the second-frame image data to obtain first-frame display data corresponding to the first-frame image data and second-frame display data corresponding to the second-frame image data, and sequentially draw the first-frame display data and the second-frame display data on a first-frame display layer and a second-frame display layer of a target display canvas; wherein, the first-frame display layer is the upper layer of the second-frame display layer;

[0008] When the first-frame display data of the first-frame display layer has been analyzed and completed based on the screen algorithm, and when pushing the second-frame display data to the user, the first-frame display data and the algorithm result data corresponding to the first-frame display data are respectively drawn on the first-frame display layer and the result display layer, so that the user can view the first-frame display data and the algorithm result data on the display screen; wherein, the result display layer is located at the topmost layer of all the display layers of the target display canvas, the first-frame display layer is the next display layer of the result display layer, and the algorithm result data is obtained by analyzing the first-frame display data based on the screen algorithm.

[0009] Optionally, if the image data to be processed further includes third-frame image data, the method further includes:

[0010] After drawing the first-frame display data on the first-frame display layer, asynchronously input the first-frame display data into the screen algorithm to analyze the first-frame display data;

[0011] Successively draw the second-frame display data on the second-frame display layer, and draw the third-frame display data corresponding to the third-frame image data on the third-frame display layer; wherein, the third-frame image data is the next-frame image data of the second-frame image data;

[0012] If the analysis of the first-frame display data is completed to obtain the algorithm result data, perform the step of respectively displaying the first-frame display layer and the result display layer on the display screen.

[0013] Optionally, the obtaining of the algorithm result data includes:

[0014] Input the first-frame display data into the screen recognition algorithm for processing to obtain the processed result data;

[0015] Analyze the processed result data based on the alarm processing algorithm to obtain the image recognition data corresponding to the first-frame display data;

[0016] Judge whether there is an alarm signal in the image recognition data;

[0017] If there is the alarm signal in the image recognition data, mark the area corresponding to the alarm signal in the image recognition data to obtain the alarm identification data.

[0018] Optionally, the respectively drawing on the first-frame display layer and the result display layer includes:

[0019] Draw the first-frame display data on the first-frame display layer, draw the image recognition data on the information display layer, and draw the alarm identification data on the identification display layer; wherein, the identification display layer is located at the topmost layer of all display layers of the target display canvas, the information display layer is the next display layer below the identification display layer, and the first-frame display layer is the next display layer below the information display layer;

[0020] Perform frame encoding on the first-frame display layer, the information display layer, and the identification display layer, so that the first-frame display data, the image recognition data, and the alarm identification data after frame encoding are displayed on the display screen.

[0021] Optionally, the step of annotating the area corresponding to the alarm signal in the image recognition data to obtain alarm identification data includes:

[0022] Determine the alarm identification position and alarm identification size of the alarm signal;

[0023] Extract the image recognition data corresponding to the alarm identification position and the alarm identification size to obtain the alarm data to be identified;

[0024] Determine the alarm screen corresponding to the alarm data to be identified in the first-frame display data, and draw it at the edge of the alarm screen to obtain the alarm identification data.

[0025] Optionally, the image processing device is connected to an audio device, and the method further includes:

[0026] Obtain the sound data collected by the audio device;

[0027] Determine the time stamp corresponding to the image data to be processed, and encode the sound data and all display data according to the time stamp to obtain audio-video data; wherein, the time stamp is used to maintain the audio-visual synchronization of the sound data and all display data, and all display data at least includes the first-frame display data and the second-frame display data;

[0028] Push the audio-video data to the client, so that the user can view the audio-video data on the client.

[0029] Optionally, after obtaining the audio-video data, the method further includes:

[0030] Convert the audio-video data into an audio-video file in a target format, and segment the audio-video file according to date and time conditions to obtain audio-video files corresponding to different dates and times; wherein, all audio-video data is stored on the server;

[0031] Receive the audio and video playback request initiated by the client; wherein, the audio and video playback request includes a target specified date;

[0032] Search on the server according to the target specified date, and determine the playback audio and video data corresponding to the target specified date;

[0033] Perform real-time streaming of the playback audio and video data, so that the user can watch the playback audio and video data on the client.

[0034] A second aspect of the embodiments of the present application provides an image processing system, including:

[0035] An acquisition unit, configured to acquire the image data to be processed collected by the imaging device; wherein, the image data to be processed includes at least first-frame image data and second-frame image data, and the second-frame image data is the next-frame image data of the first-frame image data;

[0036] The acquisition unit is further configured to perform format conversion on the first-frame image data and the second-frame image data to obtain first-frame display data corresponding to the first-frame image data and second-frame display data corresponding to the second-frame image data, and sequentially draw the first-frame display data and the second-frame display data on the first-frame display layer and the second-frame display layer of the target display canvas; wherein, the first-frame display layer is the upper layer of the second-frame display layer;

[0037] A drawing unit, configured to, when the first-frame display data of the first-frame display layer has been analyzed and completed based on the screen algorithm, and when pushing the second-frame display data to the user, draw the first-frame display data and the algorithm result data corresponding to the first-frame display data on the first-frame display layer and the result display layer respectively, so that the user can watch the first-frame display data and the algorithm result data on the display screen; wherein, the result display layer is located at the top layer of all display layers of the target display canvas, the first-frame display layer is the next display layer of the result display layer, and the algorithm result data is obtained by analyzing the first-frame display data based on the screen algorithm.

[0038] The image processing system provided in the second aspect of the embodiments of the present application is used to execute the image processing method described in the first aspect.

[0039] A third aspect of the embodiments of the present application provides an image processing device, including:

[0040] A central processing unit, a memory, an input / output interface, a wired or wireless network interface, and a power supply;

[0041] The memory is a transient storage memory or a persistent storage memory;

[0042] The central processing unit is configured to communicate with the memory and execute the instruction operations in the memory to execute the image processing method described in the first aspect.

[0043] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, which includes instructions that, when run on a computer, cause the computer to execute the image processing method described in the first aspect.

[0044] A fifth aspect of the embodiments of the present application provides a computer program product, which includes instructions that, when run on a computer, cause the computer to execute the image processing method described in the first aspect.

[0045] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages: Through an image processing method disclosed in the embodiments of the present application, first obtain the image data to be processed collected by the imaging device; wherein, the image data to be processed at least includes the first frame of image data and the second frame of image data, and the second frame of image data is the next frame of the first frame of image data; then convert the formats of the first frame of image data and the second frame of image data to obtain the first frame of display data corresponding to the first frame of image data and the second frame of display data corresponding to the second frame of image data, and sequentially draw the first frame of display data and the second frame of display data on the first frame display layer and the second frame display layer of the target display canvas; wherein, the first frame display layer is the upper layer of the second frame display layer; finally, when the first frame of display data of the first frame display layer has been analyzed and completed based on the frame algorithm, and when the second frame of display data is pushed to the user, draw the first frame of display data and the algorithm result data corresponding to the first frame of display data on the first frame display layer and the result display layer respectively, so that the user can view the first frame of display data and the algorithm result data on the display screen; wherein, the result display layer is located at the top layer of all the display layers of the target display canvas, the first frame display layer is the next display layer of the result display layer, and the algorithm result data is obtained by analyzing the first frame of display data based on the frame algorithm. Thus, by making the data displayed on the screen correspond one-to-one with the results obtained by analyzing the frame algorithm, the accuracy of image processing is achieved. And the misjudgment of the user is reduced as much as possible to reduce the probability of causing dangerous accidents. Description of the Drawings

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0047] Figure 1 Schematic diagram of a software framework for image processing disclosed in an embodiment of the present application;

[0048] Figure 2 Schematic diagram of the architecture process of an image processing system disclosed in an embodiment of the present application;

[0049] Figure 3 Schematic diagram of the process of an image processing method disclosed in an embodiment of the present application;

[0050] Figure 4 Schematic diagram of the process of another image processing method disclosed in an embodiment of the present application;

[0051] Figure 5 Schematic diagram of the process of another image processing method disclosed in an embodiment of the present application;

[0052] Figure 6 Timing diagram of a picture and an algorithm disclosed in an embodiment of the present application;

[0053] Figure 7 Element structure diagram of a picture layer disclosed in an embodiment of the present application;

[0054] Figure 8 Schematic diagram of a picture of a danger signal disclosed in an embodiment of the present application;

[0055] Figure 9 Schematic diagram of the structure of an image processing system disclosed in an embodiment of the present application;

[0056] Figure 10 Schematic diagram of the structure of an image processing device disclosed in an embodiment of the present application. Detailed implementation manners

[0057] Terms such as "first", "second", "third", "fourth", etc. (if any) in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments described here can be implemented in an order different from that shown or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0058] It should be noted that the descriptions involving "first", "second", etc. in this application are only for descriptive purposes, and should not be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one such feature. Additionally, the technical solutions between various embodiments may be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0059] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts fall within the scope of protection of this application.

[0060] For ease of understanding, please refer to Figure 1 , Figure 1 , which is a schematic diagram of a software framework for image processing disclosed in the embodiments of this application.

[0061] As can be seen from Figure 1 , the software framework in this application is generally divided into four parts. 1. Hardware acquisition layer; 2. Algorithm layer; 3. ECS engine layer; 4. Business interface layer. Among them, the ECS engine layer includes Entity, Component, and System. At the same time, the software framework in the technical solution of this application adopts a mature ECS engine architecture, ensuring that the software has excellent maintainability and scalability. At the same time, due to the good compatibility of ECS, different hardware, algorithms, and third-party dependency libraries (Third Library) can be dynamically connected to the software system in the form of components, reducing the cost of upgrade and iteration while ensuring compatibility.

[0062] Furthermore, as can be seen from Figure 1As described, in the overall framework, the User Layer includes a Camera Interface, an algorithm Interface, a livestream Interface, a File Interface, etc. The ECS includes multiple components, such as a Camera Component, a Mic Component, an Algorithm Component, an Encoder Component, an Image Layer Component, a Warning Component, a LiveStream Component, etc., which will not be elaborated here. The hardware includes a Camera hardware (USB Camera), a Mic, or a Graphics Processing Unit (GPU), etc. The algorithm layer includes various algorithms, such as Adas or QR Scan, etc. The third-party dependent libraries at least include a real-time communication technology WebRTC, a barcode processing library Zxing, or a media processor Media Processor, etc. Briefly understood, the image processing device described in the embodiments of this application can be understood as a device including an algorithm layer and an ECS engine layer. For the convenience of understanding, a detailed description will be given later.

[0063] To solve the above-mentioned technical problems and describe in detail Figure 1 the working mode of the software system, please refer to Figure 3 , Figure 3 which is a schematic flowchart of an image processing method disclosed in the embodiments of this application. It includes Step 301 - Step 303.

[0064] 301. Obtain the image data to be processed collected by the imaging device.

[0065] It should be noted in advance that this embodiment is applied to an image processing device, where the image processing device can be understood as an integrated device including an ECS engine layer or an algorithm layer as Figure 1 shown. Among them, the image processing device is connected to the imaging device. It is not difficult to understand that the imaging device can specifically be the USB Camera described above. The USB Camera and the image processing device can be connected through a USB protocol. For the convenience of understanding, it will not be elaborated here later.

[0066] Specifically, the image processing device can acquire the image data to be processed collected by the imaging device. It is not difficult to understand that the image data to be processed includes at least the first frame of image data and the second frame of image data, and the second frame of image data is the next frame of the first frame of image data. It should be added that the above first frame and second frame are not limitations on the number of frames of the image data, nor on the sequence of the frame images. In this embodiment, for the convenience of description, the first frame and the second frame are defined as two consecutive frames of image. Specifically, the first frame of image data is the image data first acquired by the imaging device, and the second frame of image data is the image data of the next frame of the first frame of image data.

[0067] Furthermore, in this embodiment, there may also be the third frame of image data or the fourth frame of image data, etc., which will not be described in detail here. Among them, the third frame of image data or the fourth frame of image data, etc. are all located after the first frame of image data and the second frame of image data.

[0068] It should also be added that in this embodiment, the imaging device can acquire the camera screen in real time and split the camera screen in the form of video frames, so as to obtain the first frame of image data and the second frame of image data described above.

[0069] 302. Convert the formats of the first frame of image data and the second frame of image data to obtain the first frame of display data corresponding to the first frame of image data and the second frame of display data corresponding to the second frame of image data, and draw the first frame of display data and the second frame of display data on the first frame display layer and the second frame display layer of the target display canvas in sequence.

[0070] Thus, convert the file formats of the first frame of image data and the second frame of image data to obtain the first frame of display data corresponding to the first frame of image data and the second frame of display data corresponding to the second frame of image data. Then, draw the first frame of display data and the second frame of display data on the first frame display layer and the second frame display layer of the target display canvas in sequence. It is not difficult to understand that in the entire display layer, the first frame display layer is the layer above the second frame display layer.

[0071] In one specific embodiment, the camera component at the algorithm layer can convert the first frame of image data and the second frame of image data into texture data that can be processed by the GPU, that is, the first frame of display data and the second frame of display data described above. Then, draw the first frame of display data and the second frame of display data corresponding to the first frame display layer and the second frame display layer in sequence.

[0072] It is not difficult to understand that in one of the achievable technical solutions, when the camera screen is captured, the captured image data (which can be understood as the first-frame image data at this time and needs to be converted into the first-frame display data through formatting) is first drawn onto layer 0 (i.e., the first-frame display layer). At this time, layer 0 is not temporarily pushed to the user for viewing. Then, the next frame of the screen (which can be understood as the second-frame image data at this time and needs to be converted into the second-frame display data through formatting) is drawn onto layer 1, and it is not given to the user for viewing either. And so on, until it is drawn onto layer X (where X can represent the number of layers that the camera component in the image processing device can handle at this time, and specific limitations are not made here).

[0073] 303. When the first-frame display data of the first-frame display layer has been analyzed and completed based on the screen algorithm, and when the second-frame display data is pushed to the user, the first-frame display data and the algorithm result data corresponding to the first-frame display data are respectively drawn on the first-frame display layer and the result display layer, so that the user can view the first-frame display data and the algorithm result data on the display screen.

[0074] Thus, when the first-frame display data of the first-frame display layer has been analyzed and completed based on the screen algorithm, and when the second-frame display data is pushed to the user, the first-frame display data and the algorithm result data corresponding to the first-frame display data can be respectively drawn on the first-frame display layer and the result display layer, so that the user can view the first-frame display data and the algorithm result data on the display screen. It is not difficult to understand that the result display layer is located at the topmost layer of all the display layers of the target display canvas, the first-frame display layer is the next display layer of the result display layer, and the algorithm result data is obtained by analyzing the first-frame display data based on the screen algorithm.

[0075] In one specific embodiment, when performing step 302, that is, when drawing the first-frame display data onto layer 0 again, the screen corresponding to layer 0 is also input into the screen algorithm for analysis. Thus, in the subsequent drawing process, if the first-frame display data has been analyzed and completed based on the screen algorithm, the algorithm processing result after algorithm analysis, that is, the algorithm result data and the screen of layer 0 (i.e., the first-frame display data), can be respectively drawn on the first-frame display layer and the result display layer. It is not difficult to understand that if the screen of layer 0 has been drawn in step 302 at this time, there is no need to draw it again, and specific details are not elaborated here. Thus, the algorithm result data and the screen corresponding to the first-frame display data of layer 0 can be displayed on the display screen, so that the user can view it on the display screen. At this time, the screen and the algorithm result are accurately corresponding.

[0076] According to the above description, in another implementable technical solution, that is, when the first-frame display data has been drawn on the first-frame display layer, the algorithm result data corresponding to the first-frame display data can be drawn on the result display layer so that the user can view the first-frame display data and the algorithm result data on the display screen. Specific details are not elaborated here.

[0077] Furthermore, for the second-frame display data, the picture of the second-frame display data can be displayed after the first-frame display data is displayed, and so on.

[0078] Through an image processing method disclosed in this embodiment, first obtain the image data to be processed collected by the imaging device; then convert the formats of the first-frame image data and the second-frame image data to obtain the first-frame display data corresponding to the first-frame image data and the second-frame display data corresponding to the second-frame image data, and sequentially draw the first-frame display data and the second-frame display data on the first-frame display layer and the second-frame display layer of the target display canvas; finally, when the first-frame display data on the first-frame display layer has been analyzed and completed based on the picture algorithm, and when the second-frame display data is pushed to the user, draw the first-frame display data and the algorithm result data corresponding to the first-frame display data on the first-frame display layer and the result display layer respectively, so that the user can view the first-frame display data and the algorithm result data on the display screen. Thus, by making the data displayed on the screen correspond one-to-one with the results obtained by analyzing the picture algorithm, the accuracy of image processing is achieved. And the misjudgment of the user is reduced as much as possible to reduce the probability of causing dangerous accidents.

[0079] When the alarm processing component recognizes various danger signals, the traditional alarm system will frame the picture area with lines and display it for the user to view. Since this type of system is usually deployed at a relatively far position from the detection target in order to cover a larger area, the detected danger area is relatively small. Therefore, the way of displaying danger signals in this solution is not easy to attract the user's attention and is easy to cause the signal to be ignored, thus resulting in safety accidents. For the convenience of Figure 3 detailed description of the above-described image processing method and to solve the technical problems described above, please refer to Figure 4 , Figure 4 which is a schematic flowchart of another image processing method disclosed in the embodiments of the present application. It includes step 401-step 410.

[0080] 401. Obtain the image data to be processed collected by the imaging device.

[0081] 402. Convert the formats of the first-frame image data and the second-frame image data to obtain the first-frame display data corresponding to the first-frame image data and the second-frame display data corresponding to the second-frame image data.

[0082] In this embodiment, steps 401 - 402 are similar to steps 301 - 302 in the foregoing Figure 3 and will not be elaborated here in detail. However, it should be noted that in this embodiment, the imaging device will collect the image data to be processed according to the set acquisition method. For example, it can be at a frequency of 30 frames per second, or at a frequency of 60 frames per second, or at a frequency of 120 frames per second. There is no specific limitation here. It should also be noted that while collecting the image data, sound data can also be collected through the Mic device. For details, reference can be made to Figure 5 the embodiment shown, which will not be described here. This embodiment will take the acquisition frequency of 30 frames per second as an example for illustration.

[0083] Furthermore, in this embodiment, multiple layers are set on the image processing device. For example, a frame layer for processing the picture, an information layer for processing information such as alarms, etc. For ease of understanding, details will be described later.

[0084] For ease of understanding and description, reference can be made to Figure 2 shown, that is, the display data of different frames is converted into texture data that can be processed by the GPU, that is, the first frame display data or the second frame display data described above. If there is third frame image data or fourth frame image data, or even other frame image data, it can also be correspondingly converted into third frame display data or fourth frame display data, or even other frame display data. There is no specific limitation here.

[0085] 403. After the first frame display data is drawn on the first frame display layer, the first frame display data is asynchronously input to the picture algorithm for analysis of the first frame display data.

[0086] Thus, corresponding to step 402, if there is third frame image data (which can be simply understood as other frame image data except the first frame image data or the second frame image data, that is, other frame image data in the image data to be processed), then after obtaining the first frame display data, the first frame display data can be drawn on the first frame display layer, and then the first frame display data is asynchronously input to the picture algorithm for analysis of the first frame display data.

[0087] In one specific embodiment, when the picture captured or recorded by the imaging device is collected, the picture will first be drawn on layer 0 (Frame 0Layer). Layer 0 will not be temporarily pushed to the user for viewing, and then step 404 is executed. At the same time, the picture corresponding to layer 0 will be sent to the algorithm for asynchronous processing. That is to say, when the picture corresponding to layer 0, that is, the first frame display data, is drawn on layer 0, the picture algorithm can also synchronously analyze the first frame display data, so as to execute step 405 (analysis step).

[0088] 404. Draw the second-frame display data on the second-frame display layer in sequence, and draw the third-frame display data corresponding to the third-frame image data on the third-frame display layer.

[0089] After the first-frame display data is drawn on the first-frame display layer, the second-frame display data can be drawn on the second-frame display layer in sequence, and the third-frame display data corresponding to the third-frame image data can be drawn on the third-frame display layer. It is not difficult to understand that the third-frame image data is the next-frame image data of the second-frame image data.

[0090] In one specific embodiment, the second-frame display data is drawn on the second-frame display layer first, and then the third-frame display data is drawn on the third-frame display layer. That is to say, draw the display data of the current frame first, and then draw the display data of the next frame.

[0091] Furthermore, in this embodiment, the frame layer (Frame Layer) is set to have layer 0 (Frame0Layer) to layer 14 (Frame 14Layer). That is to say, after layer 0 is drawn, the next-frame picture is then drawn onto layer 1 (Frame 1Layer) without showing it to the user, and so on until layer 14 (Frame14Layer) is drawn. It is not difficult to understand that layers 0 to 14 are used to cache the most recent 15 frames of the camera to form a circular queue.

[0092] For ease of understanding the drawing process of the first-frame display data, the second-frame display data, the third-frame display data, or other frame display data, please refer to Figure 6 , Figure 6 which is a timing diagram of the picture and algorithm disclosed in the embodiment of the present application. As can be seen from Figure 6 this, the camera device collects at a frequency of 30 frames per second. Therefore, in layers 0, 1 up to layer 5, the time interval between two adjacent layers is 33.3 ms.

[0093] 405. Input the first-frame display data into the picture recognition algorithm for processing to obtain processed result data, and analyze the processed result data based on the alarm processing algorithm to obtain image recognition data corresponding to the first-frame display data.

[0094] Thus, after step 402, the first-frame display data will be input into the picture recognition algorithm for processing to obtain processed result data, and then the processed result data will be analyzed based on the alarm processing algorithm to obtain image recognition data corresponding to the first-frame display data.

[0095] In one specific embodiment, the first frame of display data collected by the imaging device can be input into the image recognition algorithm component (Algorithm Component) for processing to obtain the processed result data. Then, the warning processing component (Warning Component) analyzes the algorithm results in real time to obtain the image recognition data corresponding to the first frame of display data.

[0096] Furthermore, after obtaining the image data, step 409 can be executed. Specifically, the image recognition data is drawn on the information display layer. That is, the recognition information corresponding to the first frame of display data is drawn onto the information layer (InfoLayer). This will be described in detail later.

[0097] 406. Determine whether there is an alarm signal in the image recognition data.

[0098] Based on step 405, it is necessary to determine whether there is an alarm signal in the image recognition data.

[0099] In one specific embodiment, the warning processing component (Warning Component) can synchronously determine whether an alarm signal such as drowning or a danger signal has occurred in the image recognition data. Specifically, it can be determined by detecting whether some identifiers in the image recognition data trigger the recognition identifiers of the set alarm signals. For example, the height at which a person sinks into the water, or the distance between a sharp object and a person, etc. There is no specific limitation here.

[0100] 407. When there is an alarm signal in the image recognition data, determine the alarm identifier position and alarm identifier size of the alarm signal.

[0101] Thus, when there is an alarm signal in the image recognition data, it is necessary to mark the area corresponding to the alarm signal in the image recognition data to obtain the alarm identifier data. Specifically, the alarm identifier position and alarm identifier size in the alarm signal can be determined.

[0102] In one specific embodiment, when the warning processing component recognizes an alarm signal, according to the signal position and size, the corresponding part of the picture information is extracted to determine the alarm identifier position and alarm identifier size.

[0103] 408. Extract the image recognition data corresponding to the alarm identifier position and alarm identifier size to obtain the alarm data to be marked, and determine the alarm picture corresponding to the alarm data to be marked in the first frame of display data, and draw it at the edge of the alarm picture to obtain the alarm identifier data.

[0104] Accordingly, based on step 407, image recognition data corresponding to the alarm identification position and the alarm identification size is extracted to obtain the alarm data to be identified, and the alarm screen corresponding to the alarm data to be identified in the first frame display data is determined for drawing at the edge of the alarm screen to obtain the alarm identification data.

[0105] In one specific embodiment, the image recognition data where the alarm identification position and the alarm identification size are located is extracted, and according to the identification position and the identification size, the alarm screen corresponding to the identification position and the identification size in the first frame display data is determined, magnified and covered on the top layer of the screen, and at the same time, a prominent color is drawn at the edge of the dangerous screen, thereby obtaining the alarm identification data.

[0106] For the convenience of understanding and description, please refer to Figure 8 , Figure 8 which is a schematic diagram of the screen of a danger signal disclosed in the embodiment of the present application. As Figure 8 shown, in this embodiment, after the alarm signal related to drowning is recognized, the identification position and the identification size related to drowning can be determined, and the corresponding alarm screen in the first frame display data can be found. Then, the alarm screen is magnified and a prominent color is drawn. Thus, the relevant alarm identification data is obtained.

[0107] 409. Draw the first frame display data on the first frame display layer, draw the image recognition data on the information display layer, and draw the alarm identification data on the identification display layer.

[0108] Then, the first frame display data can be drawn on the first frame display layer, the image recognition data can be drawn on the information display layer, and the alarm identification data can be drawn on the identification display layer. It is not difficult to understand that the identification display layer is on the top layer of all the display layers of the target display canvas, the information display layer is the next display layer of the identification display layer, and the first frame display layer is the next display layer of the information display layer.

[0109] In one specific embodiment, the alarm processing component draws the recognition information (image recognition data) on the information display layer (i.e., the information layer Info Layer described above), and at the same time, magnifies and highlights the above-mentioned dangerous area on the identification display layer (i.e., the picture-in-picture layer PIP Layer described above). At the same time, the first frame display data has been drawn on the first frame display layer (Frame 0Layer).

[0110] For the convenience of understanding the specific distribution of each layer, please refer to Figure 7 Figure 7 which is an element structure diagram of a screen layer disclosed in the embodiment of the present application. As Figure 7It can be seen that the topmost layer is the PIP Layer, followed by the Info Layer, Frame0Layer, Frame 1Layer, ..., Frame 14Layer. From Figure 7 and combined with Figure 8 it can be seen that the alarm or danger signal screen can be displayed at any prominent position.

[0111] 410. Perform screen encoding on the first frame display layer, information display layer, and identification display layer so that the encoded first frame display data, image recognition data, and alarm identification data are displayed on the display screen.

[0112] After the above drawing is completed, the first frame display layer, information display layer, and identification display layer can be screen-encoded, so that the encoded first frame display data, image recognition data, and alarm identification data are displayed on the display screen.

[0113] In one specific embodiment, the Image Layer Component draws the above three layers onto the Canvas and simultaneously notifies the Encoder Component to perform screen encoding. Thus, the encoded first frame display data, image recognition data, and alarm identification data can be displayed on the display screen. For example, for the drowning alarm signal, refer to Figure 8 as shown. Details are not elaborated here.

[0114] Furthermore, specifically refer to Figure 6 , assuming that layer 0 has been processed by the algorithm, then push the algorithm processing results (i.e., image recognition data and alarm identification data) and the screen of layer 0 (i.e., the first frame display data) to the user for viewing. At this time, the screen corresponds accurately to the algorithm results. At the same time, the screens of layers 1 - 14 are sequentially sent into the algorithm for processing. When layer 1 is to be pushed to the user for viewing, the algorithm processing results of layer 0 can be displayed simultaneously.

[0115] Furthermore, please continue to refer to Figure 6 , from Figure 6It can be seen that since the frame rate is 33.3 ms per frame during screen rendering, but the algorithm processing is time-consuming, assume it is 66.6 ms per frame. In this case, first send Layer 0 to the algorithm for processing. After waiting for 66.6 ms, get the algorithm result of Layer 0 and display it to the user together with Layer 0, that is, display it on the display screen. Because the algorithm itself is relatively slow, it can only calculate Layer 2 immediately after finishing calculating Layer 0, and discard the processing of Layer 1. There is no corresponding algorithm processing result for Layer 1. Only after displaying Layer 0, can Layer 1 and the algorithm result of Layer 0 be displayed to the user together. Thus, when displaying Layer 0 and Layer 1 (i.e., the first frame of display data and the second frame of display data), only the algorithm result corresponding to the first frame of display data (algorithm result 0) can be displayed. And so on, when displaying Layer 2 and Layer 3 (i.e., the third frame of display data and the fourth frame of display data), only the algorithm result corresponding to the third frame of display data (algorithm result 2) can be displayed. Specifically, the subsequent content will not be elaborated here. Until the frame display data corresponding to Layer 14 is displayed. That is to say, Layer 0 to Layer 14 form a circular queue and the executed steps are the same. At the same time, the frame display data corresponding to Layer 0 - Layer 14 will not be loaded simultaneously, but will be loaded alternately. That is, the current is Layer 0, and the next frame switches to Layer 1, and so on in a cycle. Further, it can also be understood that, for example, after displaying the screens of Layer 0 and Layer 1, the screens of Layer 0 and Layer 1 can be cleared, and then the screens corresponding to after Layer 14 are loaded, that is, the sixteenth frame of display data and the seventeenth frame of display data, thus forming a cycle. Specifically, the details will not be elaborated here one by one.

[0116] Through an image processing method disclosed in this embodiment, it is ensured that the alarm information and the screen information correspond to each other as much as possible, and at the same time, the smoothness of the screen is not affected. Meanwhile, when the alarm processing component recognizes a dangerous signal, according to the signal position and size, the corresponding part of the screen information is extracted, enlarged and covered on the top layer of the screen, and at the same time, a prominent color is drawn at the edge of the dangerous screen. The dangerous signal screen can be displayed at any prominent position. Thus, the signal focusing processing of the dangerous area can be realized, and the processing rate of the dangerous signal can be improved.

[0117] Furthermore, for the requirement of simultaneously pushing audio and video data to the server and saving it to the hard disk, the traditional solution uses two encoders. One encoder encodes the audio and video data for pushing, and the other encoder encodes the data for saving to the hard disk. Using two encoders in this solution will increase resource consumption, resulting in an increase in device power consumption and affecting the algorithm processing speed. Thus, the image processing device in this embodiment can also synchronously process the sound. Please refer to Figure 5 , Figure 5 which is a schematic flowchart of another image processing method disclosed in the embodiments of the present application. It includes Step 501 - Step 505.

[0118] 501. Obtain the sound data collected by the audio device, determine the time stamp corresponding to the image data to be processed, and encode the sound data and all display data according to the time stamp to obtain audio-visual data.

[0119] It is not difficult to understand that in this embodiment, the image processing device is also connected to the audio device. Among them, the audio device can be understood as the Mic Componeent described above.

[0120] Specifically, the image processing device obtains the sound data collected by the audio device. At the same time, it is also necessary to determine the time stamp corresponding to the image data to be processed, and encode the sound data and all display data according to the time stamp to obtain audio-visual data. It is not difficult to understand that the time stamp is used to ensure the audio-visual synchronization of the sound data and all display data. All display data includes at least the first frame of display data and the second frame of display data.

[0121] In one specific embodiment, as described above Figure 4 When the encoder encodes the picture, it will also receive the sound data collected by the microphone component (Mic Component). Among them, since it is necessary to ensure that the sound data and the display data are in a synchronous state, it is necessary to determine the time stamps of all sound data and all display data. Thus, when the encoder encodes the picture, it will also encode the sound data collected by the Mic Componeent, and ensure the synchronization of sound and picture through the time stamp, so as to obtain audio-visual data. Among them, all display data includes at least the first frame of display data and the second frame of display data. As Figure 4 described, it will also include the third frame of display data, etc.

[0122] 502. Push the audio-visual data to the client so that the user can watch the audio-visual data on the client.

[0123] Then, the audio-visual data can be pushed to the client, so that the user can watch the audio-visual data on the client.

[0124] In one specific embodiment, the encoded audio-visual data is first sent to the LiveStream Component, which pushes the stream to the server for distribution to users for viewing. It is not difficult to understand that the LiveStream Component is located in the system of the ECS engine layer. Thus, in this embodiment, the encoder can encode the picture data on the Frame layer, and then transmit the encoded picture data to WebRTC (i.e., the LiveStream Component) or the audio-visual multiplexer Muxer respectively, so as to achieve pure picture playback or audio-visual playback. It is not difficult to understand that the picture data transmitted to WebRTC can be transmitted to the broadcast network Network for playback. The audio-visual data transmitted to the video multiplexer Muxer can be transmitted to the storage space Storage for storage for subsequent call. For details, please refer to step 503-step 505.

[0125] It should also be noted that in this embodiment, the above-described picture data can also be transmitted to Storage, and the audio-visual data can also be transmitted to Network. There is no specific limitation here.

[0126] 503. Convert the audio-visual data into an audio-visual file in a target format, and split the audio-visual file according to the date and time conditions to obtain audio-visual files corresponding to different dates and times.

[0127] Thus, based on step 501, the audio-visual data will also be converted into an audio-visual file in a target format, and the audio-visual file will be split according to the date and time conditions to obtain audio-visual files corresponding to different dates and times. It is not difficult to understand that all audio-visual data is stored on the server.

[0128] In one specific embodiment, in the encoder, the audio-visual data will be converted into an audio-visual file that can be played by standard software or devices, where the format of the audio-visual file is the target format. For example, it can be the MP4 format or the MOV format, etc. There is no specific limitation here. Then, it can be split according to the period and time conditions of the audio-visual file to obtain audio-visual files with different dates and times.

[0129] Furthermore, the date and time conditions can be year, month, and day, or hour, minute, and second, etc. There is no specific limitation here. Thus, audio-visual files with different dates and times can be obtained according to the actual splitting conditions. For example, January 1, 2024 or January 2, 2024. It can also be 12:12 on January 1 or 12:13 on January 2, etc. There is no specific limitation here.

[0130] 504. Receive the audio and video playback request initiated by the client, search on the server according to the target specified date, and determine the playback audio and video data corresponding to the target specified date.

[0131] Thus, after the audio and video files of different date and time are stored on the server, the audio and video playback request initiated by the client can be received in real time, so as to search on the server according to the target specified date and determine the playback audio and video data corresponding to the target specified date. It is not difficult to understand that the audio and video playback request includes the target specified date. In this embodiment, the target specified date is used to indicate the audio and video file of the called date and time.

[0132] In one specific embodiment, when the client receives the audio and video playback request initiated by the user and requests to watch the playback. Thus, the software system can find the audio and video file of the specified date in the server and determine that the audio and video file is the playback audio and video data corresponding to the target specified date.

[0133] Furthermore, the client can query the video file of a specific time point through the server. Specifically, by clicking on the specific time search control on the client or inputting the specific time point or inputting the relevant file type, etc., the video file to be played back can be found. Thus, according to the search requirements, the client generates an audio and video playback request. In this way, the playback audio and video file required by the user can be determined according to the above method.

[0134] 505. Perform real-time streaming of the playback audio and video data so that the user can watch the playback audio and video data on the client.

[0135] Thus, based on step 504, the playback audio and video data can be streamed in real time, so that the user can watch the playback audio and video data on the client.

[0136] In one specific embodiment, after the server confirms the completion of the audio and video data to be played back, a specific component, such as WebRTC, can be used. At this time, WebRTC can initiate audio and video streaming. Thus, the client can view the audio and video picture. It is not difficult to understand that WebRTC can also be understood as the LiveStream Component described above.

[0137] It is not difficult to understand that the playback audio and video data in this embodiment can also be sound data (i.e., a sound file containing only sound), or picture display data (i.e., a picture file containing only picture display without sound), etc. For the convenience of understanding, this embodiment only takes the audio and video file as an example, but it is not a limitation on the requested file or the stored file in this embodiment.

[0138] An image processing method disclosed in this embodiment can push audio-visual data to a server and save it to a hard disk simultaneously by using an encoder, thus realizing the reuse of the encoder, improving the encoding speed, reducing resource occupation, and enhancing the feasibility of the solution.

[0139] Combined with the above Figure 1 、 Figure 3 、 Figure 4 and Figure 5 shown embodiments, for a concise description of the architecture process of the embodiments of this application, please refer to Figure 2 , Figure 2 which is a schematic diagram of the architecture process of an image processing system disclosed in the embodiments of this application.

[0140] As can be seen from Figure 2 the specific implementation steps of the image processing system in the embodiments of this application are as follows: 1. First, connect to the Camera hardware (USB Camera) through the USB protocol to obtain the camera screen in real time. 2. The obtained screen data is sent to the image recognition algorithm component (Algorithm Component) for processing to obtain the processing result. 3. The alarm processing component (Warning Component) analyzes the algorithm result in real time, draws the recognition information on the information layer (Info Layer), and at the same time determines whether dangerous signals such as drowning occur, and magnifies and highlights the dangerous area on the picture-in-picture layer (PIP Layer). 4. While performing steps 3 and 4, it is also necessary to convert the screen data into texture data that can be processed by the GPU and draw it on the frame layer (Frame Layer). 5. The image layer processing component (Image Layer Component) draws the above three layers on the canvas (Canvas), and at the same time notifies the encoding component (Encoder Component) to perform screen encoding. 6. While the encoder encodes the screen, it also encodes the sound data collected by the microphone component (Mic Component), and ensures the synchronization of sound and screen through timestamps. 7. The encoded audio-visual data is first sent to the live streaming component (LiveStream Component), pushed to the server and distributed to users for viewing. At the same time, the data is packaged into file formats such as MP4 as needed and written to the hard disk for storage for subsequent review. 8. When the client requests to watch the replay, the software system finds the file of the specified date, reads the audio-visual data, and again pushes it to the server through the live streaming component (LiveStream Component) and distributes it to users for viewing.

[0141] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this document, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of the steps or stages in other steps or other steps.

[0142] Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of an image processing system disclosed in an embodiment of the present application.

[0143] An acquisition unit 901 is configured to acquire to-be-processed image data collected by an imaging device; wherein, the to-be-processed image data at least includes first-frame image data and second-frame image data, and the second-frame image data is the next-frame image data of the first-frame image data;

[0144] The acquisition unit 901 is further configured to perform format conversion on the first-frame image data and the second-frame image data to obtain first-frame display data corresponding to the first-frame image data and second-frame display data corresponding to the second-frame image data, and sequentially draw the first-frame display data and the second-frame display data on a first-frame display layer and a second-frame display layer of a target display canvas; wherein, the first-frame display layer is the upper layer of the second-frame display layer;

[0145] A drawing unit 902 is configured to, when the first-frame display data of the first-frame display layer has been analyzed and completed based on a screen algorithm and when pushing the second-frame display data to a user, draw the first-frame display data and algorithm result data corresponding to the first-frame display data on the first-frame display layer and a result display layer respectively, so that the user can view the first-frame display data and the algorithm result data on a display screen; wherein, the result display layer is the top layer of all display layers of the target display canvas, the first-frame display layer is the next display layer of the result display layer, and the algorithm result data is obtained by analyzing the first-frame display data based on the screen algorithm.

[0146] Exemplarily, when the to-be-processed image data further includes third-frame image data, the system further includes: an input unit 903;

[0147] An input unit 903, configured to asynchronously input the first frame of display data to a picture algorithm for analysis after the first frame of display data is drawn on the first frame display layer;

[0148] The drawing unit 902 is further configured to sequentially draw the second frame of display data on the second frame display layer and draw the third frame of display data corresponding to the third frame of image data on the third frame display layer; wherein, the third frame of image data is the next frame of image data of the second frame of image data;

[0149] The obtaining unit 901 is further configured to, when the analysis of the first frame of display data is completed, obtain algorithm result data and perform the steps of respectively displaying the first frame display layer and the result display layer on the display screen.

[0150] Exemplarily, the system further includes: a judgment unit 904;

[0151] The obtaining unit 901 is specifically configured to input the first frame of display data into a picture recognition algorithm for processing to obtain processed result data;

[0152] The obtaining unit 901 is further configured to analyze the processed result data based on an alarm processing algorithm to obtain image recognition data corresponding to the first frame of display data;

[0153] The judgment unit 904 is configured to judge whether there is an alarm signal in the image recognition data;

[0154] The obtaining unit 901 is further configured to, when there is an alarm signal in the image recognition data, label the area corresponding to the alarm signal in the image recognition data to obtain alarm identification data.

[0155] Exemplarily, the system further includes: an encoding unit 905;

[0156] The drawing unit 902 is specifically configured to draw the first frame of display data on the first frame display layer, draw the image recognition data on the information display layer, and draw the alarm identification data on the identification display layer; wherein, the identification display layer is on the topmost layer of all display layers of the target display canvas, the information display layer is the next display layer below the identification display layer, and the first frame display layer is the next display layer below the information display layer;

[0157] The encoding unit 905 is configured to perform picture encoding on the first frame display layer, the information display layer, and the identification display layer so that the first frame of display data, the image recognition data, and the alarm identification data after picture encoding are displayed on the display screen.

[0158] Exemplarily, the system further includes: a determination unit 906 and an extraction unit 907;

[0159] A determination unit 906, configured to determine the alarm identification position and the alarm identification size of the alarm signal;

[0160] An extraction unit 907, configured to extract image recognition data corresponding to the alarm identification position and the alarm identification size to obtain the alarm data to be identified;

[0161] The determination unit 906 is further configured to determine an alarm screen corresponding to the alarm data to be identified in the first frame of display data, and draw it at the edge of the alarm screen to obtain alarm identification data.

[0162] Exemplarily, the system further includes: a push unit 908;

[0163] The acquisition unit 901 is further configured to acquire sound data collected by an audio device;

[0164] The determination unit 906 is further configured to determine a timestamp corresponding to the image data to be processed, and encode the sound data and all display data according to the timestamp to obtain audio-visual data; wherein, the timestamp is used to maintain the lip-sync of the sound data and all display data, and all display data includes at least the first frame of display data and the second frame of display data;

[0165] The push unit 908 is further configured to push the audio-visual data to the client so that the user can view the audio-visual data on the client.

[0166] Exemplarily, the system further includes: a receiving unit 909;

[0167] The acquisition unit 901 is further configured to convert the audio-visual data into an audio-visual file in a target format, and split the audio-visual file according to date and time conditions to obtain audio-visual files corresponding to different dates and times; wherein, all audio-visual data is stored on the server;

[0168] The receiving unit 909 is configured to receive an audio-visual playback request initiated by the client; wherein, the audio-visual playback request includes a target specified date;

[0169] The determination unit 906 is further configured to search on the server according to the target specified date to determine the playback audio-visual data corresponding to the target specified date;

[0170] The push unit 908 is further configured to perform real-time streaming of the playback audio-visual data so that the user can view the playback audio-visual data on the client.

[0171] Please refer to Figure 10 , the structural schematic diagram of an image processing apparatus disclosed in an embodiment of the present application includes:

[0172] A central processing unit 1001, a memory 1005, an input / output interface 1004, a wired or wireless network interface 1003, and a power supply 1002;

[0173] The memory 1005 is a transient memory or a persistent memory;

[0174] The central processing unit 1001 is configured to communicate with the memory 1005 and execute the instruction operations in the memory 1005 to execute the image processing method in any one of the foregoing embodiments 3 to Figure 5 as shown in any of them.

[0175] An embodiment of the present application further provides a chip system, which is characterized in that the chip system includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the image processing method in any one of the foregoing embodiments 3 to Figure 5 as shown in any of them.

[0176] An embodiment of the present application further provides a computer-readable storage medium, the computer-readable storage medium includes instructions, when the instructions run on a computer, the computer is caused to execute the image processing method in any one of the foregoing embodiments 3 to Figure 5 as shown in any of them.

[0177] An embodiment of the present application further provides a computer program product containing instructions, when the computer program product runs on a computer, the computer is caused to execute the image processing method in any one of the foregoing embodiments 3 to Figure 5 as shown in any of them.

[0178] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0179] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in an electrical, mechanical, or other form.

[0180] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] In addition, each functional unit in various embodiments of the present application may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

Claims

1. An image processing method, characterized in that: Applied to an image processing device, the image processing device is connected to a camera device, and the method comprises: Acquire the image data to be processed collected by the camera device; wherein the image data to be processed includes at least a first frame of image data and a second frame of image data, and the second frame of image data is the next frame of image data of the first frame of image data; Convert the formats of the first frame image data and the second frame image data to obtain first frame display data corresponding to the first frame image data and second frame display data corresponding to the second frame image data, and draw the first frame display data and the second frame display data in sequence on a first frame display layer and a second frame display layer of a target display canvas; wherein the first frame display layer is a layer above the second frame display layer; When the first frame display data of the first frame display layer has been analyzed based on the picture algorithm, and when the second frame display data is pushed to the user, the algorithm result data corresponding to the first frame display data will be drawn on the result display layer, so that the user can view the first frame display data and the algorithm result data on the display screen; wherein, the result display layer is located at the topmost layer of all display layers of the target display canvas, the first frame display layer is the next display layer of the result display layer, and the algorithm result data is obtained after analyzing the first frame display data based on the picture algorithm.

2. The image processing method according to claim 1, characterized in that: If the image data to be processed further includes a third frame of image data, the method further includes: After drawing the first frame display data on the first frame display layer, asynchronously inputting the first frame display data into the picture algorithm to analyze the first frame display data; sequentially drawing the second frame display data on the second frame display layer, and drawing the third frame display data corresponding to the third frame image data on the third frame display layer; wherein the third frame image data is the next frame image data of the second frame image data; If the first frame display data has been analyzed, the algorithm result data is obtained, and the steps of displaying the first frame display layer and the result display layer on the display screen are performed.

3. The image processing method according to claim 1, characterized in that: The algorithm result data obtained includes: Inputting the first frame display data into a picture recognition algorithm for processing to obtain processing result data; Analyze the processing result data based on an alarm processing algorithm to obtain image recognition data corresponding to the first frame display data; Determining whether there is an alarm signal in the image recognition data; If the alarm signal exists in the image recognition data, an area in the image recognition data corresponding to the alarm signal is marked to obtain alarm identification data.

4. The image processing method according to claim 3, characterized in that: The drawing is performed on the first frame display layer and the result display layer respectively, including: Draw the first frame display data on the first frame display layer, draw the image recognition data on the information display layer, and draw the alarm identification data on the identification display layer; wherein the identification display layer is located at the topmost layer of all display layers of the target display canvas, the information display layer is the next display layer of the identification display layer, and the first frame display layer is the next display layer of the information display layer; The first frame display layer, the information display layer and the identification display layer are screen-encoded so that the first frame display data, the image recognition data and the alarm identification data after the screen encoding are displayed on the display screen.

5. The image processing method according to claim 3, characterized in that: The step of marking the area in the image recognition data corresponding to the alarm signal to obtain the alarm identification data includes: Determining the alarm mark position and the alarm mark size of the alarm signal; Extracting the image recognition data corresponding to the position of the alarm mark and the size of the alarm mark to obtain the alarm data to be marked; An alarm picture corresponding to the alarm data to be identified in the first frame display data is determined, and drawing is performed on the edge of the alarm picture to obtain the alarm identification data.

6. The image processing method according to claim 1, characterized in that: The image processing device is connected to an audio device, and the method further includes: Acquiring sound data collected by the audio device; Determine a timestamp corresponding to the image data to be processed, and encode the sound data and all display data according to the timestamp to obtain audio and video data; wherein the timestamp is used to ensure audio and video synchronization between the sound data and all display data, and the all display data at least includes the first frame display data and the second frame display data; The audio and video data are pushed to a client so that the user can watch the audio and video data on the client.

7. The image processing method according to claim 6, characterized in that: After obtaining the audio and video data, the method further includes: Convert the audio and video data into audio and video files of a target format, and segment the audio and video files according to date and time conditions to obtain audio and video files corresponding to different dates and times; wherein all audio and video data are stored on the server; Receiving an audio and video playback request initiated by the client; wherein the audio and video playback request includes a target specified date; Searching the server according to the target specified date to determine the playback audio and video data corresponding to the target specified date; The playback audio and video data is streamed in real time so that the user can watch the playback audio and video data on the client.

8. An image processing system, characterized in that: Applied to an image processing device, the image processing device is connected to a camera device, and the system comprises: An acquisition unit, used for acquiring the image data to be processed collected by the camera device; wherein the image data to be processed includes at least a first frame of image data and a second frame of image data, and the second frame of image data is a next frame of image data of the first frame of image data; The acquisition unit is further used to perform format conversion on the first frame image data and the second frame image data to obtain first frame display data corresponding to the first frame image data and second frame display data corresponding to the second frame image data, respectively, and to draw the first frame display data and the second frame display data in sequence on a first frame display layer and a second frame display layer of a target display canvas; wherein the first frame display layer is a layer above the second frame display layer; A drawing unit, for drawing the first frame display data and the algorithm result data corresponding to the first frame display data on the first frame display layer and the result display layer respectively when the first frame display data of the first frame display layer has been analyzed based on the picture algorithm and when the second frame display data is pushed to the user, so that the user can view the first frame display data and the algorithm result data on the display screen; wherein the result display layer is located at the topmost layer of all display layers of the target display canvas, the first frame display layer is the next display layer of the result display layer, and the algorithm result data is obtained after analyzing the first frame display data based on the picture algorithm.

9. An image processing device, characterized in that: The device comprises: CPU, memory, input and output interfaces, wired or wireless network interfaces, and power supply; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions, and when the instructions are executed on a computer, the computer is enabled to perform the image processing method according to any one of claims 1 to 7.

Images