Video object detection and SEI information embedding method and device for front-end intelligent processing

Abstract

The invention relates to the technical field of computers, in particular to a video object detection and SEI information embedding method and device for front-end intelligent processing. The method comprises the following steps: acquiring a corresponding real-time front-end video image based on front-end image acquisition equipment, inputting the real-time front-end video image into a built-in image signal processor and a neural network reasoning unit, and acquiring corresponding detection data; packaging the detection data into SEI information of the video code stream according to a preset format, and obtaining a corresponding target video code stream; and sending the target video code stream to a receiving end based on a standard streaming media protocol, controlling the receiving end to continuously decode the target video code stream and extract corresponding object detection data and a decoded image in the SEI information, and executing an operation on the decoded image based on the object detection information. According to the method and the device, the system efficiency and the response speed are ensured, the system architecture is simplified, and the consumption of computing resources is reduced in the prior art.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for video object detection and SEI information embedding of front-end intelligent processing. Background Art

[0002] Currently, video encoding based on the H.264 / H.265 standard is widely used in streaming media transmission. At the same time, AI-based image object detection technology has important applications in video surveillance, intelligent analysis and other fields.

[0003] The traditional approach is to deploy AI detection tasks on back-end servers. Although this approach can centrally process large amounts of data, it requires transmitting a large amount of raw video data to the server for processing, which not only consumes a lot of computing resources but also causes system response delays. To improve this, some technologies have attempted to transmit AI detection results through additional network protocols, but this has increased the complexity of the system.

[0004] From the above, we can see that how to simplify the system architecture and reduce the consumption of computing resources while ensuring system efficiency and response speed remains to be solved. Summary of the invention

[0005] The present application provides a method, device, electronic device and storage medium for front-end intelligent processing of video object detection and SEI information embedding, which can solve the problem of ensuring system efficiency and response speed while simplifying system architecture and reducing computing resource consumption in related technologies. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for video object detection and SEI information embedding for front-end intelligent processing includes: obtaining a corresponding real-time front-end video image based on a front-end image acquisition device, inputting the real-time front-end video image into a built-in image signal processor and a neural network inference unit, and obtaining corresponding detection data, wherein the detection data includes but is not limited to human faces, human shapes, human joints, motorcycles, bicycles, electric vehicles, cars, car license plates, location information, and credibility; encapsulating the detection data into the SEI information of a video bitstream in a preset format to obtain a corresponding target video bitstream, wherein the format of the target video bitstream remains unchanged; sending the target video bitstream to a receiving end based on a standard streaming media protocol, controlling the receiving end to continue decoding the target video bitstream and extracting the corresponding object detection data and decoded image in the SEI information, and performing operations on the decoded image based on the object detection information.

[0007] In an exemplary embodiment, the method further includes: the detection data embedded in the SEI information complies with the SEI standard format of H.264 / H.265 encoding, and the receiving end can extract the corresponding object detection data according to the SEI standard format.

[0008] In an exemplary embodiment, in the process of inputting the real-time front-end video image into a built-in image signal processor and a neural network inference unit to obtain corresponding detection data, the method also includes: the neural network inference unit is able to run multiple models or a multifunctional model simultaneously, and regularly update the neural network inference unit and the image signal processor.

[0009] In an exemplary embodiment, in the process of encapsulating the detection data into the SEI information of the video code stream in a preset format, the method also includes: before the target video code stream is transmitted, calling a predefined encryption algorithm and a key, converting the detection data into a binary form, encrypting the detection data converted into the binary form as an input of the encryption algorithm, and embedding the encrypted detection data into the SEI information; after the target video code stream is transmitted at the receiving end, the encrypted SEI information is decrypted based on the encryption algorithm and the key to restore it to unencrypted SEI information.

[0010] In an exemplary embodiment, the method further includes: after the device is started, loading the preset alarm trigger conditions and preventive measures configuration; calling the corresponding real-time front-end video image, and detecting the real-time front-end video image to determine whether the alarm trigger conditions are met; if the alarm trigger conditions are met, starting the corresponding alarm mechanism.

[0011] In an exemplary embodiment, the method also includes: when the front-end device is started, the version information corresponding to the neural network inference unit currently in use is loaded, and when the front-end device monitors the update notification of the receiving end through the network, the front-end device parses the update information in the update notification, wherein the update information includes the version number, download link and verification information of the new model; based on the update information, the new neural network inference unit is downloaded from the front-end device, and an integrity check is performed during the download process; after the new neural network inference unit is downloaded, the new neural network inference unit is installed to the specified path, and the version information in the new neural network inference unit is updated, and after the installation is completed, the front-end device switches to the new neural network inference unit.

[0012] According to one aspect of the present application, a video object detection and SEI information embedding device for front-end intelligent processing includes:

[0013] A detection data acquisition module, which acquires corresponding real-time front-end video images based on the front-end image acquisition device, and inputs the real-time front-end video images into a built-in image signal processor and a neural network reasoning unit to acquire corresponding detection data, wherein the detection data includes but is not limited to human faces, human shapes, human joints, motorcycles, bicycles, electric vehicles, cars, car license plates, location information, and credibility;

[0014] An encapsulation module, used for encapsulating the detection data into SEI information of a video bitstream according to a preset format, and obtaining a corresponding target video bitstream, wherein the format of the target video bitstream remains unchanged;

[0015] The extraction module sends the target video code stream to the receiving end based on the standard streaming protocol, controls the receiving end to continue decoding the target video code stream and use it to extract the corresponding object detection data and decoded image in the SEI information, and performs operations on the decoded image based on the object detection information.

[0016] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the front-end intelligent processing video object detection and SEI information embedding method as described above.

[0017] According to one aspect of the present application, a storage medium stores computer-readable instructions thereon, and the computer-readable instructions are executed by one or more processors to implement the video object detection and SEI information embedding method of front-end intelligent processing as described above.

[0018] According to one aspect of the present application, a computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the front-end intelligent processing video object detection and SEI information embedding method as described above.

[0019] The beneficial effects of the technical solution provided by the present application are as follows: by prepending the object detection task to the front-end device and embedding the detection result into the SEI information of the video code stream, high efficiency of video object detection and SEI information embedding is achieved; the intelligent processing of the front-end device reduces the load on the back-end server and improves the system response speed, while utilizing standard video encoding and streaming media transmission protocols to ensure the compatibility and transmission efficiency of the system, which not only improves the performance of the video intelligent analysis system, but also simplifies the system architecture and reduces the consumption of computing resources while ensuring system efficiency and response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0021] Figure 1 It is a schematic diagram of the implementation environment involved in this application;

[0022] Figure 2 is a flow chart of a method for video object detection and SEI information embedding by front-end intelligent processing according to an exemplary embodiment;

[0023] Figure 3 is a flowchart from S101 to S103 in another method for video object detection and SEI information embedding in front-end intelligent processing according to an exemplary embodiment;

[0024] Figure 4 is a flowchart from S104 to S106 in another method for video object detection and SEI information embedding in front-end intelligent processing according to an exemplary embodiment;

[0025] Figure 5 It is a structural block diagram of a video object detection and SEI information embedding device for front-end intelligent processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0027] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0028] The method for video object detection and SEI information embedding for front-end intelligent processing provided in the present application can effectively improve the accuracy of video object detection and SEI information embedding for front-end intelligent processing. Accordingly, the method for video object detection and SEI information embedding for front-end intelligent processing is suitable for a video object detection and SEI information embedding device for front-end intelligent processing, and the video object detection and SEI information embedding device for front-end intelligent processing can be deployed in an electronic device.

[0029] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0030] Figure 1 The schematic diagram of the implementation environment involved in a front-end intelligent processing video object detection and SEI information embedding method includes a collection end and a service end.

[0031] See also Figure 2 The embodiment of the present application provides a method for video object detection and SEI information embedding by front-end intelligent processing, which is applicable to electronic devices, which may be Figure 1 The server side in the implementation environment is shown.

[0032] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0033] like Figure 2 As shown, the method may include the following steps:

[0034] S100, based on the front-end image acquisition device, the corresponding real-time front-end video image is obtained, and the real-time front-end video image is input into the built-in image signal processor and neural network inference unit to obtain the corresponding detection data.

[0035] The detection data includes but is not limited to human faces, human shapes, human joints, motorcycles, bicycles, electric vehicles, cars, car license plates, location information, and credibility; the neural network reasoning unit (TPU) is a high-performance hardware accelerator specifically used to perform neural network reasoning tasks.

[0036] Front-end image acquisition devices (such as cameras) are responsible for real-time video image acquisition. These devices usually have high-definition video acquisition capabilities and can automatically adjust exposure and white balance under different lighting conditions to ensure image quality. By acquiring high-quality real-time video images, they provide a basic data source for subsequent image processing and object detection, and the real-time performance ensures that the system can respond to and process changes in the current scene in a timely manner.

[0037] The built-in image signal processor (ISP) is required to pre-process the collected real-time video images, such as denoising, color correction, contrast enhancement, etc., to optimize the image quality, and input the pre-processed real-time video images into the neural network inference unit (TPU) for real-time object detection. The TPU uses a pre-trained model to perform inference analysis on the image, identify the target object, and generate detection data including location information, credibility, etc. The TPU performs real-time inference, reducing dependence on the back-end server and improving the system response speed.

[0038] It should be pointed out here that in the embodiment of the present application, the neural network inference unit can run multiple models or a multifunctional model simultaneously, and regularly update the neural network inference unit and the image signal processor.

[0039] By running multiple models, more accurate detection can be performed for different target objects, for example, faces and vehicles can be detected simultaneously in the same frame of an image, improving the versatility and detection accuracy of the system.

[0040] In addition to running multiple models, TPU can also run a multifunctional model, which is usually a large neural network that can detect multiple target objects at the same time, but requires maintaining one multifunctional model instead of multiple independent models, which simplifies the model management process; a multifunctional model can handle multiple target objects, improving the versatility and adaptability of the system.

[0041] With the development of technology and changes in application scenarios, the neural network reasoning unit (TPU) and image signal processor (ISP) need to be updated regularly. By regularly updating models and algorithms, the accuracy and robustness of detection can be continuously improved. The updated TPU and ISP can better support new application scenarios, such as new target object detection, and ensure continuous optimization and maintenance of the system, extending the service life of the equipment.

[0042] S110, encapsulating the detection data into SEI information of the video bitstream according to a preset format, and obtaining a corresponding target video bitstream.

[0043] The format of the target video bitstream remains unchanged; the detection data is encapsulated into the supplementary enhancement information (SEI information) of the H.264 / H.265 video bitstream according to a predefined format. The SEI information is part of the video bitstream and can carry additional information without changing the basic format of the video bitstream. The encapsulated data includes but is not limited to the position, size, credibility and attribute data of each detected object.

[0044] It should be pointed out here that the detection results are transmitted synchronously with the video data through SEI information, without the need for new network protocols, which can simplify the system architecture. At the same time, the video bitstream format remains unchanged, ensuring seamless integration with existing decoders and streaming media transmission systems. The receiving end does not need to re-perform complex neural network inference to obtain the detection results.

[0045] S120, sending the target video code stream to the receiving end based on the standard streaming protocol, controlling the receiving end to continue decoding the target video code stream and extracting the corresponding object detection data and decoded image in the SEI information, and performing operations on the decoded image based on the object detection information.

[0046] Among them, the target video code stream embedded with detection information is sent to the receiving end using standard streaming media transmission protocols (such as RTSP, RTMP, HTTP Live Streaming, etc.). On the one hand, the standardized transmission protocol ensures compatibility with existing streaming media transmission systems and reduces development and maintenance costs. On the other hand, the transmission process is stable and reliable, reducing the possibility of data loss.

[0047] In addition, the receiving end receives the target video code stream through the standard streaming protocol and decodes it. After decoding, the object detection data is extracted from the SEI information, and the decoded image can be further intelligently analyzed and processed based on the extracted object detection data; the receiving end can quickly obtain the detection results and perform accurate image processing, such as object cutout, etc.; at the same time, the computational burden of the receiving end is reduced, and the overall performance of the system is improved.

[0048] By executing the above steps S100 to S120, the response speed and efficiency of the system are improved, the system architecture is simplified, and the compatibility and security of the system are enhanced.

[0049] In the embodiment of the present application, when the front-end device is started, Figure 3 As shown, the method further continues with the following steps:

[0050] S101, when the front-end device is started, the version information corresponding to the neural network inference unit currently in use is loaded, and when the front-end device monitors the update notification of the receiving end through the network, the front-end device parses the update information in the update notification.

[0051] Among them, the update information includes the version number, download link and verification information of the new model; it can ensure that the front-end device can receive the latest update information in time, realize real-time update of the model, and the front-end device automatically parses the update information, reducing the need for manual intervention and improving update efficiency.

[0052] S102, starting to download a new neural network inference unit from the front-end device based on the update information, and performing integrity verification during the downloading process.

[0053] During the download process, the front-end device will perform an integrity check on the downloaded data, usually using a hash algorithm (such as MD5 or SHA256) to ensure that the data has not been tampered with. If the check fails, it will be downloaded again until it succeeds. This ensures that the downloaded new model file is complete and has not been tampered with, improves the security of the system, reduces update failures caused by data corruption, and increases the success rate of updates.

[0054] S103, after the new neural network inference unit is downloaded, the new neural network inference unit is installed to the specified path, and the version information in the new neural network inference unit is updated. After the installation is completed, the front-end device switches to the new neural network inference unit.

[0055] Among them, after the download is complete, the front-end device will install the new model file to the specified path; during the installation process, the system will update the version information to ensure the consistency of version management; after the installation is complete, the system will verify the validity of the new model file and make necessary configuration changes, and the front-end device needs to switch to the new neural network inference unit.

[0056] Through the above process, the front-end device can automatically receive and update the neural network inference unit to ensure that the system is always kept up to date. Through regular updates, the front-end device can continuously improve the detection accuracy and adapt to new application scenarios, which not only improves the update efficiency and security of the system, but also enhances the maintainability and adaptability of the system.

[0057] In addition, the detection data to be transmitted is encrypted before transmission, which specifically includes the following process:

[0058] S111, before the target video code stream is transmitted, a predefined encryption algorithm and key are retrieved, the detection data is converted into a binary form, the detection data converted into the binary form is used as an input of the encryption algorithm for encryption, and the encrypted detection data is embedded in the SEI information.

[0059] By using encryption algorithms and keys, encryption algorithms can provide data integrity verification to ensure that data has not been tampered with during transmission, thereby ensuring data security during transmission.

[0060] Among them, the detection data usually contains text and numerical information, which needs to be converted into binary form before encryption. Converting different types of detection data into binary form can facilitate subsequent encryption operations. At the same time, binary data can be seamlessly transmitted on different systems and platforms.

[0061] The converted binary detection data is used as the input of the encryption algorithm for encryption. The encryption process generates encrypted data, usually a new binary sequence, and the encrypted data cannot be directly read during transmission, thereby ensuring the confidentiality of the data.

[0062] In addition, the encrypted detection data is embedded into the SEI information of the video bitstream according to a predefined format, and the SEI information is a part of the video bitstream and can carry additional information without affecting the decoding of the video itself.

[0063] S112, after receiving the target video code stream, the encrypted SEI information is decrypted based on the encryption algorithm and the key to restore it to unencrypted SEI information.

[0064] It should be pointed out here that restoring the decrypted data to the original detection data can ensure that only the receiving end with the correct key can decrypt the data, thereby improving the security of the system.

[0065] Through the execution steps from S111 to S112 above, not only the security of data transmission is improved, but also the integrity of the data and the compatibility of the system are ensured, and through encryption processing, the communication between the front-end device and the receiving end becomes more secure and reliable.

[0066] At the same time, if Figure 4 As shown, the method in the embodiment of the present application also includes:

[0067] S104, after the device is started, the preset alarm triggering conditions and preventive measures configuration are loaded.

[0068] Among them, the preventive measures configuration is usually stored in the memory of the device, which can be in the form of a configuration file or a database. The alarm triggering conditions include alarm triggering conditions (such as the appearance of specific objects, credibility thresholds, etc.) and corresponding preventive measures (such as sounding an alarm, sending a notification, triggering an external device, etc.).

[0069] By automatically loading configuration information, manual intervention is reduced and the system's automation level is improved. In addition, the configuration information can be updated at any time according to actual conditions to adapt to different application scenarios.

[0070] S105, retrieve the corresponding real-time front-end video image, detect the real-time front-end video image and determine whether the alarm triggering condition is met.

[0071] Among them, the real-time front-end video image is preprocessed using the built-in image signal processor (ISP) to optimize the image quality, and the preprocessed real-time front-end video image is input into the neural network reasoning unit (TPU) for real-time object detection.

[0072] Analyzing the detection results to determine whether the preset alarm trigger conditions are met can ensure that the system can monitor the video stream in real time and detect potential abnormal situations in a timely manner.

[0073] S106: If the alarm triggering condition is met, start the corresponding alarm mechanism.

[0074] When the detection results meet the alarm triggering conditions, the front-end device activates the corresponding alarm mechanism, which may include but is not limited to: issuing sound alarms (such as beeps, voice prompts, etc.); flashing lights (such as LED lights); triggering external devices (such as alarm systems connected to front-end devices); recording event logs to mark the time, location and related information of specific events; automatically taking snapshots or recording short video clips, and saving them to local storage media or sending them to a designated address (such as email, cloud storage, etc.).

[0075] Through the execution steps from S104 to S106 above, not only the response speed and security of the system are improved, but also the automated management and flexibility of the system are strengthened; through the preset alarm trigger conditions and preventive measures configuration, the front-end equipment can effectively monitor the video stream and take action when necessary to ensure that potential abnormal situations are discovered and handled in a timely manner.

[0076] The following is an embodiment of the device of the present application, which can be used to perform the video object detection and SEI information embedding method for front-end intelligent processing involved in the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of the video object detection and SEI information embedding method for front-end intelligent processing involved in the present application.

[0077] See also Figure 5 In the embodiment of the present application, a video object detection and SEI information embedding device for front-end intelligent processing is provided, including but not limited to:

[0078] The detection data acquisition module 200 acquires the corresponding real-time front-end video image based on the front-end image acquisition device, and inputs the real-time front-end video image into the built-in image signal processor and the neural network reasoning unit to acquire the corresponding detection data, wherein the detection data includes but is not limited to human face, human shape, human joints, motorcycles, bicycles, electric vehicles, cars, car license plates, location information, and credibility;

[0079] The encapsulation module 210 is used to encapsulate the detection data into the SEI information of the video bitstream according to a preset format, and obtain the corresponding target video bitstream, wherein the format of the target video bitstream remains unchanged;

[0080] The extraction module 200 sends the target video code stream to the receiving end based on the standard streaming protocol, controls the receiving end to continue decoding the target video code stream and use it to extract the corresponding object detection data and decoded image in the SEI information, and performs operations on the decoded image based on the object detection information.

[0081] In an exemplary embodiment, the device further includes but is not limited to:

[0082] The detection data embedded in the SEI information complies with the SEI standard format of H.264 / H.265 encoding, and the receiving end can extract the corresponding object detection data according to the SEI standard format.

[0083] In an exemplary embodiment, the device further includes but is not limited to:

[0084] The neural network inference unit is capable of running multiple models or a multi-function model simultaneously and periodically updating the neural network inference unit and the image signal processor.

[0085] In an exemplary embodiment, the device further includes but is not limited to:

[0086] The encryption module, before the target video code stream is transmitted, calls a predefined encryption algorithm and a key, converts the detection data into a binary form, encrypts the detection data converted into the binary form as an input of the encryption algorithm, and embeds the encrypted detection data into the SEI information;

[0087] The decryption module, after receiving the target video code stream at the receiving end, is used to decrypt the encrypted SEI information based on the encryption algorithm and the key and restore it to the unencrypted SEI information.

[0088] In an exemplary embodiment, the device further includes but is not limited to:

[0089] Loading module, used to load preset alarm trigger conditions and preventive measures configuration after the device is started;

[0090] The judgment module retrieves the corresponding real-time front-end video image, detects the real-time front-end video image and uses it to judge whether the alarm triggering condition is met;

[0091] The alarm mechanism startup module is used to start the corresponding alarm mechanism if the alarm triggering conditions are met.

[0092] In an exemplary embodiment, the device further includes but is not limited to:

[0093] A version information loading module, when the front-end device is started, is used to load the version information corresponding to the currently used neural network inference unit. When the front-end device monitors the update notification of the receiving end through the network, the front-end device parses the update information in the update notification, wherein the update information includes the version number, download link and verification information of the new model;

[0094] A download module, which is used to start downloading a new neural network inference unit from the front-end device based on the update information and perform integrity verification during the download process;

[0095] The installation module is used to install the new neural network inference unit to the specified path after the new neural network inference unit is downloaded, and to update the version information in the new neural network inference unit. After the installation is completed, the front-end device switches to the new neural network inference unit.

[0096] It should be noted that the front-end intelligent processing video object detection and SEI information embedding device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing front-end intelligent processing video object detection and SEI information embedding. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the front-end intelligent processing video object detection and SEI information embedding device will be divided into different functional modules to complete all or part of the functions described above.

[0097] In addition, the front-end intelligent processing video object detection and SEI information embedding device and the front-end intelligent processing video object detection and SEI information embedding method provided in the above embodiments belong to the same concept, and the specific way in which each module performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0098] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more processors, so that the electronic device implements the above-mentioned front-end intelligent processing video object detection and SEI information embedding method.

[0099] In addition, a storage medium is provided in an embodiment of the present application, on which computer-readable instructions are stored, and the computer-readable instructions are executed by one or more processors to implement the above-mentioned front-end intelligent processing video object detection and SEI information embedding method.

[0100] In an embodiment of the present application, a computer program product is provided, which includes computer-readable instructions, and the computer-readable instructions are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the above-mentioned front-end intelligent processing video object detection and SEI information embedding method.

[0101] Compared with related technologies, the method of the present application realizes video object detection and SEI information embedding for front-end intelligent processing, improving the system's response speed, data security and automated management level. Specifically, the front-end device can capture video images and perform object detection in real time, encapsulate the detection data into the SEI information of the video stream, and transmit it to the receiving end through the standard streaming media protocol. The receiving end can extract the detection data according to the SEI standard format and perform further intelligent analysis and processing. In addition, the system also has the versatility and regular update capabilities of the neural network reasoning unit, ensuring the continuous improvement and adaptability of the system.

[0102] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0103] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video object detection and SEI information embedding method with front-end intelligent processing, characterized in that: include: Based on the front-end image acquisition device, a corresponding real-time front-end video image is obtained, and the real-time front-end video image is input into a built-in image signal processor and a neural network reasoning unit to obtain corresponding detection data, wherein the detection data includes a human face, a human shape, a human joint point, a motorcycle, a bicycle, an electric vehicle, a car, a car license plate, location information, and credibility; Encapsulating the detection data into SEI information of a video bitstream in a preset format to obtain a corresponding target video bitstream, wherein the format of the target video bitstream remains unchanged; The target video code stream is sent to the receiving end based on the standard streaming media protocol, the receiving end is controlled to continue decoding the target video code stream and extract the corresponding object detection data and decoded image in the SEI information, and an operation is performed on the decoded image based on the object detection information.

2. The method according to claim 1, characterized in that The method further comprises: The detection data embedded in the SEI information complies with the SEI standard format of H.264 / H.265 encoding, and the receiving end can extract the corresponding object detection data according to the SEI standard format.

3. The method according to claim 2, characterized in that In the process of inputting the real-time front-end video image into a built-in image signal processor and a neural network inference unit to obtain corresponding detection data, the method further includes: The neural network inference unit is capable of running multiple models or a multifunctional model simultaneously, and regularly updating the neural network inference unit and the image signal processor.

4. The method according to claim 2, characterized in that In the process of encapsulating the detection data into the SEI information of the video code stream in a preset format, the method further includes: Before the target video code stream is transmitted, a predefined encryption algorithm and a key are retrieved, the detection data is converted into a binary form, the detection data converted into the binary form is used as an input of the encryption algorithm for encryption, and the encrypted detection data is embedded in the SEI information; After the target video code stream is received at the receiving end, the encrypted SEI information is decrypted based on the encryption algorithm and the key to restore it to the unencrypted SEI information.

5. The method according to claim 1, characterized in that The method further comprises: After the device is started, the preset alarm trigger conditions and preventive measures configuration are loaded; Retrieving the corresponding real-time front-end video image, and detecting the real-time front-end video image to determine whether the alarm triggering condition is met; If the alarm triggering conditions are met, the corresponding alarm mechanism is activated.

6. The method according to claim 1, characterized in that The method further comprises: When the front-end device is started, the version information corresponding to the currently used neural network inference unit is loaded. When the front-end device monitors the update notification of the receiving end through the network, the front-end device parses the update information in the update notification, wherein the update information includes the version number, download link and verification information of the new model; Starting to download a new neural network reasoning unit from the front-end device based on the update information, and performing integrity verification during the downloading process; After the new neural network inference unit is downloaded, the new neural network inference unit is installed to the specified path, and the version information in the new neural network inference unit is updated. After the installation is completed, the front-end device switches to the new neural network inference unit.

7. A video object detection and SEI information embedding device with front-end intelligent processing, characterized in that: include: A detection data acquisition module, which acquires corresponding real-time front-end video images based on the front-end image acquisition device, and inputs the real-time front-end video images into a built-in image signal processor and a neural network reasoning unit to acquire corresponding detection data, wherein the detection data includes human faces, human shapes, human joints, motorcycles, bicycles, electric vehicles, cars, car license plates, location information, and credibility; An encapsulation module, used for encapsulating the detection data into SEI information of a video bitstream according to a preset format, and obtaining a corresponding target video bitstream, wherein the format of the target video bitstream remains unchanged; The extraction module sends the target video code stream to the receiving end based on the standard streaming protocol, controls the receiving end to continue decoding the target video code stream and use it to extract the corresponding object detection data and decoded image in the SEI information, and performs operations on the decoded image based on the object detection information.

8. The device according to claim 7, characterized in that The device also includes: The detection data embedded in the SEI information complies with the SEI standard format of H.264 / H.265 encoding, and the receiving end can extract the corresponding object detection data according to the SEI standard format.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the video object detection and SEI information embedding method for front-end intelligent processing as described in any one of claims 1 to 6.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the video object detection and SEI information embedding method for front-end intelligent processing as described in any one of claims 1 to 6.

Images