Video analysis system, method, electronic device, and storage medium

Abstract

The application discloses a video analysis system, method, electronic equipment and storage medium. The system comprises: an application layer comprising at least one business system, wherein the business system is configured to send a video analysis instruction to a service layer when an image to be analyzed is acquired; the service layer comprises at least one server, wherein the server is configured to analyze and process the image to be analyzed to obtain a video analysis result when the video analysis instruction is received; and a data layer comprising a release channel component, wherein the release channel component is configured to release the video analysis result through at least one release channel. The above technical solution outputs the video analysis result to multiple release channels through the release channel component, realizes sharing of the video analysis result, and breaks the information island effect of the system.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a video analysis system, method, electronic device, and storage medium. Background Technology

[0002] With the development of video analytics technology, it has been widely used in coal mine safety early warning scenarios.

[0003] In the process of realizing this invention, the inventors discovered that the prior art has at least the following technical problems: after analyzing coal mine scene videos and obtaining video analysis results, the video analysis results can only be used by the current business system, resulting in the problem of information not being shared. Summary of the Invention

[0004] This invention provides a video analysis system, method, electronic device, and storage medium to solve the problem of the inability to share video information analysis results.

[0005] According to one aspect of the present invention, a video analysis system is provided, comprising:

[0006] The application layer includes at least one business system, which is used to send video analysis instructions to the service layer when an image to be analyzed is acquired;

[0007] The service layer includes at least one server, which is used to analyze and process the image to be analyzed upon receiving a video analysis instruction, and obtain video analysis results.

[0008] The data layer includes a publishing channel component, which is used to publish the video analysis results through at least one publishing channel.

[0009] According to another aspect of the present invention, a video analysis method is provided, executed by a video analysis system, the system comprising an application layer, a service layer, and a data layer; the application layer includes at least one business system, the service layer includes at least one server, and the data layer includes a publishing channel component; the method includes:

[0010] Once the image to be analyzed is obtained, the application layer business system sends video analysis instructions to the service layer.

[0011] Upon receiving a video analysis instruction from the server in the service layer, the image to be analyzed is processed to obtain the video analysis result.

[0012] The video analysis results are published through at least one publishing channel via the data layer's publishing channel component.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor;

[0015] and a memory communicatively connected to the at least one processor;

[0016] The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the video analysis method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video analysis method according to any embodiment of the present invention.

[0018] The technical solution of this invention, through the application layer business system, sends a video analysis command to the service layer when it obtains the image to be analyzed; through the service layer server, when it receives the video analysis command, it analyzes and processes the image to be analyzed to obtain the video analysis result; through the data layer publishing channel component, the video analysis result is output to multiple publishing channels, thereby realizing the sharing of video analysis results and breaking the information silo effect of the system.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of a video analysis system according to Embodiment 1 of the present invention;

[0022] Figure 2 This is a schematic diagram of the structure of a video analysis system according to Embodiment 2 of the present invention;

[0023] Figure 3 This is a schematic diagram of the structure of a video analysis system according to Embodiment 3 of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of a video analysis system according to Embodiment 4 of the present invention;

[0025] Figure 5 This is a schematic diagram of a video analysis method provided in Embodiment 5 of the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the video analysis method of this invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1 This is a schematic diagram of a video analysis system according to Embodiment 1 of the present invention. The system includes an application layer 110, a service layer 120, and a data layer 130. The application layer 110 includes at least one business system, which sends a video analysis command to the service layer upon acquiring an image to be analyzed. The service layer 120 includes at least one server, which analyzes and processes the image to be analyzed upon receiving the video analysis command to obtain video analysis results. The data layer 130 includes a publishing channel component, which publishes the video analysis results through at least one publishing channel.

[0031] In this embodiment, the application layer 110 includes one or more business systems, which are systems applied to various stages of mining. For example, the application layer 110 may include, but is not limited to, business systems such as vehicle control systems, safety monitoring and control systems, and drilling and mining systems. Each business system includes at least one image acquisition device, which can be used to acquire images of the coal mine scene where the business system is located. The image acquisition device may include, but is not limited to, fixed-position cameras, vehicle-mounted mobile devices, handheld recording devices, and smart mining lamps.

[0032] The service layer 120 is communicatively connected to the application layer 110 and may include one or more servers. For example, the service layer 120 may include, but is not limited to, streaming media servers, video application servers, and video algorithm servers, which work together to analyze and process the image to be analyzed in order to obtain video analysis results.

[0033] The data layer 130 is communicatively connected to the service layer 120. The data layer may include one or more publishing channel components. Through the publishing channel components, video analysis results can be published through multiple channels, including but not limited to MQTT, TCP / UDP, FTP server, WebSocket, HTTP / HTTPS, and audio / video streaming media.

[0034] In some embodiments, the data layer 130 further includes a database, which may be a MySQL database, an Oracle database, or the like, for storing information such as image and / or video analysis results.

[0035] Specifically, the application layer 110's business system acquires the image to be analyzed and sends video analysis instructions to the service layer. The image to be analyzed refers to the image to be subjected to safety analysis; in other words, by analyzing the image, it can be determined whether there is a hazard at the mining site. The video analysis instructions can be used to trigger the server to process the image to be analyzed. Furthermore, upon receiving the video analysis instructions, the server in the service layer analyzes the image to be analyzed and obtains the video analysis results, which include whether there are any safety hazards or not. Further, the server can send the video analysis results to the data layer, and then the data layer's publishing channel component outputs the video analysis results to multiple publishing channels, realizing the sharing of video analysis results and breaking the information silo effect of the system.

[0036] The technical solution of this invention involves the application layer's business system, upon acquiring an image to be analyzed, sending a video analysis command to the service layer; the service layer's server, upon receiving the video analysis command, analyzes and processes the image to be analyzed to obtain the video analysis result; and the data layer's publishing channel component publishes the video analysis result through at least one publishing channel. This technical solution, by outputting the video analysis result to multiple publishing channels through the publishing channel component, achieves the sharing of video analysis results and breaks down the information silo effect of the system.

[0037] Example 2

[0038] Figure 2 This is a schematic diagram of a video analysis system provided in Embodiment 2 of the present invention. The video analysis system of this embodiment can be combined with various optional solutions provided in the video analysis systems of the above embodiments. The video analysis system provided in this embodiment has been further optimized. Optionally, the image to be analyzed includes an image to be analyzed and / or a video to be analyzed; correspondingly, the business system includes: a vehicle control system 111, used to collect the image to be analyzed and / or the video to be analyzed under the vehicle control system 111; a safety monitoring system 112, used to collect the image to be analyzed and / or the video to be analyzed under the safety monitoring system 112; and a drilling and mining system 113, used to collect the image to be analyzed and / or the video to be analyzed under the drilling and mining system 113.

[0039] In this embodiment, the business system includes a vehicle control system 111, a safety monitoring system 112, and a drilling and mining system 113. The vehicle control system 111 controls the operation of mining vehicles. The safety monitoring system 112 monitors for hazards at the mining site. The drilling and mining system 113 is the mining operation system. Each of these business systems may include one or more image acquisition devices. These devices are used to acquire images and / or videos to be analyzed within the corresponding business system. All business systems access the service layer using a unified access standard, enabling multiple business systems to connect to the service layer. This allows the service layer to analyze images from different business systems, improving the adaptability of the video analysis system.

[0040] The technical solution of this invention connects the vehicle control system 111, the safety monitoring system 112, and the drilling and mining system 113 to the service layer, respectively, thereby enabling the service layer to analyze images from different business systems and improving the adaptability of the video analysis system.

[0041] Example 3

[0042] Figure 3This is a schematic diagram of a video analysis system provided in Embodiment 3 of the present invention. The video analysis system of this embodiment can be combined with various optional solutions provided in the video analysis systems of the above embodiments. The video analysis system provided in this embodiment has been further optimized. Optionally, the server includes: a streaming media server 121, used for caching the image to be analyzed; a video application server 122, used for retrieving the image to be analyzed from the streaming media server 121, performing enhancement processing on the image to be analyzed to obtain an enhanced image to be analyzed, and sending the enhanced image to be analyzed to a video algorithm server 123; the video algorithm server 123, used for analyzing and calculating the enhanced image to be analyzed to obtain video analysis results.

[0043] In this embodiment, the server includes, but is not limited to, a streaming media server 121, a video application server 122, and a video algorithm server 123. Specifically, the streaming media server 121 can cache the image to be analyzed and perform operations such as format conversion. The video application server 122 can enhance the image to be analyzed and send the enhanced image, as well as send model start instructions and analysis result instructions. The video algorithm server 123 can perform analysis operations such as image classification, object detection, image segmentation, and behavior analysis on the enhanced image to be analyzed, thereby obtaining video analysis results.

[0044] Optionally, based on the above embodiments, the streaming media server 121 is further configured to: upon receiving first protocol information sent by the application layer 110, send interface information to the application layer 110, wherein the first protocol information includes a session identifier and a data type, and the interface information includes an IP address and port information; and upon receiving the interface information, receive the image to be analyzed sent by the application layer 110.

[0045] The first protocol information pertains to the connection establishment protocol between the application layer 110 and the streaming media server 121, and may include a session identifier and data type. For example, the session identifier may be a Universally Unique Identifier (UUID); the data type may be an image or video type. The IP and port information in the interface information may be the available IP and port of the current session, or the IP and port of a WebSocket service.

[0046] For example, the application layer 110 and the streaming media server 121 can communicate using the WebSocket communication protocol. The application layer 110 can act as a client, and the streaming media server 121 can act as a server. Specifically, the application layer 110 sends first protocol information in standard JSON format to the streaming media server 121 and establishes a connection with the streaming media server 121. After receiving the first protocol information sent by the application layer 110, the streaming media server 121 can send interface information in standard JSON format to the application layer 110. After receiving the interface information, the application layer 110 can transmit the image to be analyzed to the WebSocket service link via the WebSocket communication protocol, or push the video to be analyzed to the streaming media link via the RTP protocol.

[0047] Based on the above embodiments, optionally, the streaming media server 121 is further configured to: decode the image to be analyzed when the image to be analyzed is a picture, to obtain a decoded image; convert the decoded image into a YUV image format image, and perform video encoding on the YUV image format image to obtain the target analysis video.

[0048] For example, the video encoding can be H.264. Specifically, when the streaming media server 121 receives an image to be analyzed, it simultaneously starts the WebSocket service link and the streaming media link, decodes the image to obtain the decoded image, converts the decoded image into a YUV image format, and then performs H.264 encoding on the YUV image format to obtain the target analysis video. After obtaining the target analysis video, it can be RTP encapsulated, and the encapsulated video can be packaged according to preset packaging rules. The packaged video is then pushed to the streaming media link, enabling video analysis even for images, thus improving the adaptability of the video analysis system.

[0049] Optionally, based on the above embodiments, the video application server 122 is further configured to: receive second protocol information sent by the application layer 110, wherein the second protocol information includes a model identifier, a session identifier, auxiliary parameters, a publishing channel, and a business system identifier; retrieve the image to be analyzed from the streaming media server based on the session identifier; perform image enhancement processing on the image to be analyzed to obtain an enhanced image to be analyzed, wherein the image enhancement processing includes one or more of filtering, histogram equalization, and image sharpening processing; package the enhanced image to be analyzed and send the packaged image to be analyzed to the streaming media server 121; send a model start command to the video algorithm server 123, wherein the model start command includes a model identifier; receive model start success information corresponding to the model identifier fed back by the algorithm server 123; save the auxiliary parameters corresponding to the model identifier; send the enhanced image to be analyzed to the video algorithm server 123; and send third protocol information to the application layer, wherein the third protocol information includes a model identifier, an original session identifier, an image-enhanced session identifier, and a model start result.

[0050] The second protocol information refers to the protocol sent from application layer 110 to video application server 122, which may include, but is not limited to, model identifier, session identifier, auxiliary parameters, publishing channel, and business system identifier. The third protocol information refers to the protocol sent from video application server 122 to application layer 110, which may include, but is not limited to, model identifier, original session identifier, image-enhanced session identifier, and model startup result. Additional parameters for the publishing channel are shown in Table 1.

[0051] Table 1

[0052]

[0053]

[0054] For example, upon receiving the second protocol information, the video application server 122 matches and retrieves the image to be analyzed from the streaming media server 121 based on the session identifier. This image to be analyzed can be a video stream in H.264 standard format. Further, the H.264 standard format video stream can be decoded. After decoding, the decoded video can be enhanced. Image enhancement processing includes one or more of filtering, histogram equalization, and image sharpening, which are not limited here. Then, the enhanced image to be analyzed can be packaged into an RTP stream and pushed to the streaming media server. Furthermore, through the internal protocol between the video application server 122 and the video algorithm server 123, the video application server 122 sends a model start command to the video algorithm server 123. This model start command includes a model identifier. Subsequently, the video algorithm server 123 sends back model start success information corresponding to the model identifier. The video application server 122 receives the model start success information corresponding to the model identifier from the algorithm server 123. The video application server 122 can also save the auxiliary parameters corresponding to the model identifier to the algorithm server 123. The video application server 122 can also send the enhanced image to be analyzed to the video algorithm server 123. The video application server 122 can also send third protocol information in standard JSON protocol format to the application layer 110. The third protocol information includes the model identifier, the original session identifier, the session identifier after image enhancement, and the model start result.

[0055] Based on the above embodiments, optionally, the video algorithm server 123 is further configured to perform at least one of the following operations on the enhanced image to be analyzed: image classification processing on the enhanced image to be analyzed; target detection processing on the enhanced image to be analyzed; image segmentation processing on the enhanced image to be analyzed; and behavior analysis processing on the enhanced image to be analyzed.

[0056] For example, the video application server 122 and the video algorithm server 123 transmit data through an interaction protocol. This interaction protocol is a network transmission protocol based on the KCP protocol. This protocol can solve the problem of slow network speed of the TCP protocol under network congestion and can improve the transmission speed while ensuring reliability. In this embodiment, the video algorithm server 123 acts as the server, and the video application server 122 acts as the client. The video application server 122 can send three types of instructions, as shown in Table 2.

[0057] Table 2

[0058]

[0059] NALU stands for Network Abstraction Layer Unit.

[0060] For example, after performing image classification processing on the enhanced image to be analyzed, category information, as well as auxiliary parameters such as the confidence level and alarm information of the category, can be obtained. The category information can be personnel intrusion into the driving area, personnel intrusion into the dangerous area, personnel leaving their post, etc.

[0061] Based on the above embodiments, optionally, the video application server 122 is further configured to: upon receiving the video analysis results fed back by the video algorithm server 123, determine auxiliary parameters based on the model identifier and the session identifier; determine business linkage data based on the auxiliary parameters and the video analysis results, and determine the publishing channel of the business linkage data.

[0062] For example, after receiving the video analysis results from the video algorithm server 123, the video application server 122 can match auxiliary parameters in a preset association relationship based on the model identifier and session identifier. Then, it can input the auxiliary parameters and video analysis results into the data processing model to obtain business linkage data. Based on the session identifier in the business linkage data, it matches the publishing channel and pushes the business linkage data to the publishing channel corresponding to the session identifier. The operations corresponding to each publishing channel are shown in Table 3.

[0063] Table 3

[0064]

[0065] Among them, the information sent through the five publishing channels—MQTT, TCP / UDP, FTP server, WebSocket, and HTTP / HTTPS—is encrypted. After the business system retrieves the information from the server, it needs to decrypt the information to obtain the data content. The audio and video streaming media publishing channel sends the video analysis results to the streaming media server 121 in the form of instructions. The streaming media server 121 matches the corresponding bitstream based on the session identifier and applies the analysis results to the corresponding video frames using a tagged image processing method. The video analysis results include a timestamp, analysis result, and session identifier.

[0066] Optionally, based on the above embodiments, the streaming media server 121 is further configured to: cache video data of a preset duration upon receiving the video analysis result sent by the video application server 122; and fuse the video analysis result with the video data of the preset duration to obtain video data containing the video analysis result.

[0067] For example, streaming media service 121 performs the action of caching 5 seconds of video data. After receiving the video analysis result instruction, streaming media service 121 finds the corresponding bitstream according to the session identifier; based on the time node of the instruction received, it continues to save 5 seconds of video data, saving a total of 10 seconds of video data; it traverses the 10 seconds of video data, finds the video frame corresponding to the timestamp, and tags the video analysis result to the video frame corresponding to the timestamp in the form of a tag, obtaining video data containing the video analysis result, which can be used for historical playback and alarm analysis.

[0068] In some embodiments, the business system can perform auxiliary parameter verification. Specifically, auxiliary parameter verification code is designed and compiled into a platform-supported .so dynamic library, and named with the business system identifier. To ensure that business linkage data is only recognized and used by the business system, reversible encryption can be performed. After obtaining the business linkage data, the business system can use a decryption key to decrypt the data and obtain the correct linkage data for subsequent linkage and alarm execution.

[0069] In some optional embodiments, a correspondence table between the model and auxiliary parameters can be pre-established. The correspondence table may include the model identifier, the data style of the analysis results output by the model, and the auxiliary parameters supported by the model.

[0070] The technical solution of this invention involves caching the image to be analyzed using a streaming media server; retrieving the image to be analyzed from the streaming media server using a video application server; enhancing the image to be analyzed to obtain an enhanced image to be analyzed; and sending the enhanced image to be analyzed to a video algorithm server. The video algorithm server then performs analysis and calculations on the enhanced image to obtain the video analysis result, thereby achieving automatic video analysis and improving video analysis efficiency.

[0071] Example 4

[0072] Figure 4 This is a schematic diagram of the structure of a video analysis system provided in Embodiment 4 of the present invention. The video analysis system in this embodiment is a preferred example of the above embodiments.

[0073] In this embodiment, the video analytics system includes an application layer 110, a service layer 120, and a data layer 130. The application layer 110 includes a vehicle control system, a safety monitoring system, and a drilling and mining system. The service layer 120 includes, but is not limited to, a streaming media server 121, a video application server 122, and a video algorithm server 123. The data layer 130 includes a publishing channel component 131 and a MySQL database 132.

[0074] The streaming media server 121 may include a streaming media service unit, a tag processing unit, and a communication unit. The video application server 122 includes an interface processing module, an image enhancement module, an encoding / decoding module, a video distribution module, an auxiliary parameter verification module, an internal communication module, and a publishing unit. The video algorithm server 123 includes an internal communication module, a training model unit, a decoding module, and a real-time analysis and computation unit. The publishing channel component 131 may include, but is not limited to, MQTT, TCP / UDP, FTP server, WebSocket, HTTP / HTTPS, and audio / video streaming media publishing channels.

[0075] The technical solution of this invention outputs video analysis results to multiple publishing channels through a publishing channel component, thereby achieving the sharing of video analysis results and breaking the information silo effect of the system.

[0076] Example 5

[0077] Figure 5 This is a flowchart of a video analysis method provided in Embodiment 5 of the present invention. This embodiment is applicable to situations where automatic hazard analysis of images is performed. The method can be executed by a video analysis system, which includes an application layer, a service layer, and a data layer. The application layer includes at least one business system, the service layer includes at least one server, and the data layer includes a publishing channel component. Figure 5 As shown, the method includes:

[0078] S510: Through the application layer business system, after obtaining the image to be analyzed, a video analysis command is sent to the service layer.

[0079] S520. Upon receiving a video analysis instruction from the server in the service layer, the image to be analyzed is processed to obtain the video analysis result.

[0080] S530. The video analysis results are published through at least one publishing channel via the publishing channel component of the data layer.

[0081] The technical solution of this invention involves the application layer's business system, upon acquiring an image to be analyzed, sending a video analysis command to the service layer; the service layer's server, upon receiving the video analysis command, analyzes and processes the image to be analyzed to obtain the video analysis result; and the data layer's publishing channel component publishes the video analysis result through at least one publishing channel. This technical solution, by outputting the video analysis result to multiple publishing channels through the publishing channel component, achieves the sharing of video analysis results and breaks down the information silo effect of the system.

[0082] Optionally, the image to be analyzed includes a picture to be analyzed and / or a video to be analyzed; correspondingly, the method further includes:

[0083] Collect images and / or videos to be analyzed from the vehicle control system;

[0084] Collect images and / or videos to be analyzed from the security monitoring system;

[0085] Collect images and / or videos to be analyzed from the drilling and mining system.

[0086] Optionally, the method further includes:

[0087] The image to be analyzed is cached using a streaming media server;

[0088] The video application server retrieves the image to be analyzed from the streaming media server, performs enhancement processing on the image to be analyzed to obtain an enhanced image to be analyzed, and sends the enhanced image to the video algorithm server.

[0089] The enhanced image to be analyzed is processed by a video algorithm server to obtain video analysis results.

[0090] Optionally, the method further includes:

[0091] Upon receiving the first protocol information sent by the application layer, the streaming media server sends interface information to the application layer. The first protocol information includes a session identifier and a data type, and the interface information includes an IP address and a port. Upon receiving the interface information, the application layer receives the image to be analyzed sent by the application layer.

[0092] Optionally, the method further includes:

[0093] Using the streaming media server, if the image to be analyzed is an image, the image to be analyzed is decoded to obtain a decoded image; the decoded image is converted into a YUV image format image, and the YUV image format image is video encoded to obtain the target analysis video.

[0094] Optionally, the method further includes:

[0095] The video application server receives second protocol information sent by the application layer, wherein the second protocol information includes a model identifier, a session identifier, auxiliary parameters, a publishing channel, and a business system identifier; it retrieves the image to be analyzed from the streaming media server based on the session identifier; it performs image enhancement processing on the image to be analyzed to obtain an enhanced image to be analyzed, wherein the image enhancement processing includes one or more of filtering, histogram equalization, and image sharpening; it packages the enhanced image to be analyzed and sends the packaged image to the streaming media server; it sends a model start command to the video algorithm server, wherein the model start command includes a model identifier; it receives model start success information corresponding to the model identifier from the algorithm server; it saves the auxiliary parameters corresponding to the model identifier; it sends the enhanced image to be analyzed to the video algorithm server; and it sends third protocol information to the application layer, wherein the third protocol information includes a model identifier, an original session identifier, an image-enhanced session identifier, and a model start result.

[0096] Optionally, the method further includes:

[0097] The video algorithm server performs at least one of the following operations on the enhanced image to be analyzed: image classification processing on the enhanced image to be analyzed; target detection processing on the enhanced image to be analyzed; image segmentation processing on the enhanced image to be analyzed; and behavior analysis processing on the enhanced image to be analyzed.

[0098] Optionally, the method further includes:

[0099] Upon receiving video analysis results from the video algorithm server, the video application server determines auxiliary parameters based on model identifiers and session identifiers; it then determines business linkage data based on the auxiliary parameters and video analysis results, and identifies the distribution channel for the business linkage data.

[0100] Optionally, the method further includes:

[0101] Upon receiving the video analysis results from the video application server, the streaming media server caches video data of a preset duration; then, it merges the video analysis results with the video data of the preset duration to obtain video data containing the video analysis results.

[0102] The video analysis method provided in the embodiments of the present invention can be executed by the video analysis system provided in any embodiment of the present invention, and has the corresponding beneficial effects of the video analysis system.

[0103] Example 6

[0104] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0105] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An I / O interface 15 is also connected to the bus 14.

[0106] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0107] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video analysis methods, which include:

[0108] Once the image to be analyzed is obtained, the application layer business system sends video analysis instructions to the service layer.

[0109] Upon receiving a video analysis instruction from the server in the service layer, the image to be analyzed is processed to obtain the video analysis result.

[0110] The video analysis results are published through at least one publishing channel via the data layer's publishing channel component.

[0111] In some embodiments, the video analysis method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video analysis method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video analysis method by any other suitable means (e.g., by means of firmware).

[0112] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0116] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0117] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0118] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video analysis system, characterized in that, include: The application layer includes at least one business system, which is used to send video analysis instructions to the service layer when an image to be analyzed is acquired; The service layer includes at least one server, which is used to analyze and process the image to be analyzed upon receiving a video analysis instruction, and obtain video analysis results. The data layer includes a publishing channel component, which is used to publish the video analysis results through at least one publishing channel; wherein, the publishing channel includes at least MQTT, TCP / UDP, FTP server, WebSocket, HTTP / HTTPS and audio / video streaming media; The server includes: A streaming media server is used to cache the image to be analyzed; A video application server is used to retrieve the image to be analyzed from the streaming media server, perform enhancement processing on the image to be analyzed to obtain an enhanced image to be analyzed, and send the enhanced image to be analyzed to the video algorithm server. A video algorithm server is used to perform analysis and calculation on the enhanced image to be analyzed, and to obtain video analysis results; The video application server is also used for: The application layer sends a second protocol information, which includes a model identifier, a session identifier, auxiliary parameters, a publishing channel, and a business system identifier. Based on the session identifier, the image to be analyzed is retrieved from the streaming media server; The image to be analyzed is subjected to image enhancement processing to obtain an enhanced image to be analyzed, wherein the image enhancement processing includes one or more of filtering, histogram equalization and image sharpening processing; The enhanced image to be analyzed is packaged and then sent to the streaming media server. Send a model start command to the video algorithm server, the model start command including a model identifier; Receive the model startup success information corresponding to the model identifier from the algorithm server; Save the auxiliary parameters corresponding to the model identifier; The enhanced image to be analyzed is sent to the video algorithm server; The third protocol information is sent to the application layer, wherein the third protocol information includes the model identifier, the original session identifier, the session identifier after image enhancement, and the model startup result; The video application server is also used for: Upon receiving the video analysis results from the video algorithm server, auxiliary parameters are determined based on the model identifier and session identifier. Based on the auxiliary parameters and video analysis results, business linkage data is determined, and the publication channel of the business linkage data is determined; wherein, the publication channel of the business linkage data refers to the publication channel corresponding to the session identifier.

2. The system according to claim 1, characterized in that, The images to be analyzed include pictures and / or videos to be analyzed; Accordingly, the business system includes: The vehicle control system is used to acquire images and / or videos to be analyzed under the vehicle control system. A security monitoring system is used to collect images and / or videos to be analyzed under the security monitoring system. A drilling and mining system is used to collect images and / or videos to be analyzed from the drilling and mining system.

3. The system according to claim 1, characterized in that, The streaming media server is also used for: Upon receiving the first protocol information sent by the application layer, the interface information is sent to the application layer, wherein the first protocol information includes a session identifier and a data type, and the interface information includes an IP address and a port address. Upon receiving interface information at the application layer, the system receives the image to be analyzed sent by the application layer.

4. The system according to claim 1, characterized in that, The streaming media server is also used for: If the image to be analyzed is a picture, the image to be analyzed is decoded to obtain the decoded image; The decoded image is converted into a YUV image format, and the YUV image format image is then video encoded to obtain the target analysis video.

5. The system according to claim 1, characterized in that, The video algorithm server is also used for: Perform at least one of the following operations on the enhanced image to be analyzed: The enhanced image to be analyzed is then subjected to image classification processing; The enhanced image to be analyzed is then subjected to target detection processing; The enhanced image to be analyzed is then subjected to image segmentation processing; Behavioral analysis processing is performed on the enhanced image to be analyzed.

6. The system according to claim 1, characterized in that, The streaming media server is also used for: Upon receiving the video analysis results sent by the video application server, cache video data for a preset duration. The video analysis results are fused with video data of a preset duration to obtain video data containing the video analysis results.

7. A video analysis method, characterized in that, This is performed by a video analytics system, which includes an application layer, a service layer, and a data layer. The application layer includes at least one business system, the service layer includes at least one server, and the data layer includes a publishing channel component. The method includes: Once the image to be analyzed is obtained, the application layer business system sends video analysis instructions to the service layer. Upon receiving a video analysis instruction from the server in the service layer, the image to be analyzed is processed to obtain the video analysis result. The video analysis results are published through at least one publishing channel via the data layer publishing channel component; wherein, the publishing channel includes at least MQTT, TCP / UDP, FTP server, WebSocket, HTTP / HTTPS, and audio / video streaming media. The method further includes: The image to be analyzed is cached using a streaming media server; The video application server retrieves the image to be analyzed from the streaming media server, performs enhancement processing on the image to be analyzed to obtain an enhanced image to be analyzed, and sends the enhanced image to the video algorithm server. The enhanced image to be analyzed is processed by a video algorithm server to obtain video analysis results. The method further includes: The video application server receives second protocol information sent by the application layer, wherein the second protocol information includes a model identifier, a session identifier, auxiliary parameters, a publishing channel, and a business system identifier; it retrieves the image to be analyzed from the streaming media server based on the session identifier; it performs image enhancement processing on the image to be analyzed to obtain an enhanced image to be analyzed, wherein the image enhancement processing includes one or more of filtering, histogram equalization, and image sharpening; it packages the enhanced image to be analyzed and sends the packaged image to the streaming media server; it sends a model start command to the video algorithm server, wherein the model start command includes a model identifier; it receives model start success information corresponding to the model identifier from the algorithm server; it saves the auxiliary parameters corresponding to the model identifier; it sends the enhanced image to be analyzed to the video algorithm server; and it sends third protocol information to the application layer, wherein the third protocol information includes a model identifier, an original session identifier, an image-enhanced session identifier, and a model start result. The method further includes: Upon receiving video analysis results from the video algorithm server, the video application server determines auxiliary parameters based on the model identifier and session identifier; it then determines business linkage data based on the auxiliary parameters and video analysis results, and identifies the publishing channel for the business linkage data; wherein, the publishing channel for the business linkage data refers to the publishing channel corresponding to the session identifier.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the video analysis method of claim 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video analysis method of claim 7.

Images