A method and system for edge-cloud compact video encoding supporting crowd intelligence perception

Abstract

The present invention discloses a method and system for end-edge-cloud compact video encoding supporting crowd intelligence perception. In the cloud-edge-end system, single-camera and cross-camera semantic target images are optimized, and based on target tracking and image priority, the semantic picture frame containing the largest semantic information is selected, and the picture frame sequence of the semantic target object on its time domain trajectory is feature extracted, and the obtained compact features are used for the reconstruction of the semantic target object, so as to realize the compact encoding of the semantic target of the video crowd intelligence, collect the control parameters for feedback control, focus on the area of ​​interest or relatively important semantic perception, and guide the coordinated bit rate allocation and quantization control optimization between multiple cameras, so as to realize the coordinated optimization encoding of the end-edge-cloud of the video crowd intelligence multi-camera. In the image search business for massive camera video data, the present invention efficiently encodes the massive camera video data, and supports efficient storage, transmission and analysis of the end-edge-cloud.

Description

Technical Field

[0001] The present invention belongs to the technical field of video big data application, and specifically relates to an end-edge-cloud compact video encoding method and system supporting crowd intelligence perception. Background Art

[0002] In the past decade, technologies and application formats such as smart cities, digital twins, and metaverse have emerged one after another, attracting the attention of the industry. The proposed city brain system can process large-scale multi-source data in real time, and has been deployed and implemented in scenarios such as urban traffic and police monitoring, and its effectiveness has been verified. In smart city applications, video big data collaborative group intelligence perception and collaborative analysis are key, but the application of massive video big data group intelligence perception still has some problems, facing challenges such as difficult storage, difficult retrieval, and difficult identification.

[0003] There is a lot of redundancy in the video data collected by the group intelligence video application system. How to select and extract high-quality data from low-quality redundant data is one of the key issues. Less than 10% of the video data recorded by urban cameras is reviewed by regulators. In the existing mainstream technology system, cameras independently collect compressed videos, and massive compressed video data is aggregated to the cloud. Collaborative perception analysis is performed in the cloud. This system faces the challenges of high transmission and storage costs and high costs for single-channel video intelligent analysis. Without major technological breakthroughs, tens of millions of cameras cannot achieve "full network sharing" real-time data aggregation, let alone "all-time available" network analysis and identification.

[0004] Research has shown that there is a hierarchical correspondence between the hierarchical structure of the cerebral cortex and the level of visual feature representation, revealing the similarities between human and machine vision. The HVS retina has about 120 million photoreceptor cells. The connection channel between the retinal photoreceptor cells and the brain is the optic nerve. The ratio of the number of optic nerves to retinal photoreceptor cells is about 1:126. The transmission bandwidth of photosensitive data through the visual connection channel needs to be reduced by about 126 times, corresponding to the "eye-brain" perception structure. This compression is not a simple video compression, but a multi-level visual feature extraction and compact expression.

[0005] Inspired by the HVS eye-brain perception mechanism, the city-level big data holographic perception system can be abstracted into a city-level eye-brain perception system. Massive cameras are equivalent to retinal photoreceptors, edge domains and hierarchical transmission links are equivalent to transmission neurons, and the cloud center brain is equivalent to the brain. In the city brain eye-brain perception model, massive cameras generate extremely large amounts of video data. How to extract features and compactly encode these data, realize hierarchical collaborative and efficient storage and transmission of video data, and solve the pain point of extremely high transmission and storage costs are key issues that need to be solved in the city brain video crowd intelligence perception.

[0006] From the perspective of the city brain's collective intelligence collaborative perception, to achieve collaborative and efficient encoding of massive camera data, several problems need to be solved:

[0007] (1) Multi-camera collaborative perception and compact visual coding: A semantic target is recorded and recognized by multiple visual sensors in a certain physical space and over a period of time. Multi-camera collaborative perception is required. We study efficient multi-camera semantic target de-duplication recognition and spatiotemporal coding methods to achieve accurate recognition and de-duplication description of semantic targets.

[0008] (2) Multi-camera joint video coding optimization: Traditional video coding optimization is based on a single camera independently. In the urban brain video cluster scenario, many cameras work together to provide support for the city brain's large-scale perception. For this scenario, it is necessary to study multi-camera joint coding optimization methods, including bit rate optimization allocation, to provide more efficient and intelligent coding support for the city brain's video cluster perception.

[0009] In summary, building a city brain video crowd-intelligence perception system model inspired by the eye-brain perception mechanism and studying the multi-camera collaborative joint coding optimization method can provide support for the optimization of the city brain crowd-intelligence video perception coding algorithm.

[0010] In the existing technology, a front-end intelligent video coding and analysis framework VaBUS is proposed for end-cloud video collaborative computing. The front-end background extraction and foreground semantic segmentation are studied from the perspective of semantic object fidelity, and compact encoding and efficient transmission of surveillance videos are realized for surveillance scenarios. In addition, there is a digital retina technology system proposed by analogy to the human retinal system, which evolves and innovates the traditional visual computing architecture, explores and attempts to solve the pain points of video big data, and then further proposes a digital retina model generation, utilization and transmission paradigm, and studies the multi-stream collaborative computing mode to overcome the model reuse and transmission bottlenecks in the end-edge-cloud environment, and the corresponding digital retina standard is in the process of being formulated. Multi-camera collaborative target tracking and target re-identification are a typical type of work. By correlating cross-camera data and fusing multi-camera data information, multi-target tracking and target re-identification of multiple (cross) cameras are explored.

[0011] The above-mentioned existing technologies have explored cloud-edge collaborative video encoding and representation to a certain extent. However, they have not considered the redundancy of unified semantic target objects in different cameras, and the encoding has not considered eliminating such redundancy, and have not achieved multi-objective performance optimization such as storage, transmission cost, and encoding compression. Summary of the invention

[0012] In order to solve the shortcomings of the prior art, achieve the purpose of eliminating data redundancy and improving the efficiency of data storage, transmission and encoding compression, the present invention adopts the following technical solutions:

[0013] A method for edge-cloud compact video encoding supporting crowd intelligence perception includes the following steps:

[0014] Step 1: Single camera semantic target detection and semantic target image selection;

[0015] Step 2: cross-camera semantic target image optimization: for the semantic target images of the cameras adjacent to each other in a certain range of physical space, a cross-camera semantic target optimization algorithm is used to perform association analysis, generate a cross-camera semantic target track, and select a group of semantic target images from the semantic target track;

[0016] Step 3: Compact coding of semantic targets in video crowd intelligence: Based on target tracking and picture priority, the semantic picture frame containing the largest semantic information is selected, and the picture frame sequence of the semantic target object on its time domain trajectory is extracted, and the obtained compact features are used to reconstruct the semantic target object;

[0017] Step 4: Video crowd intelligence multi-camera end-edge-cloud collaborative optimization encoding; build a feedback control link from the cloud to the edge server and then through the end-side server to the end-side camera, collect control parameters for feedback control at different granularities for feedback adjustment, so that the city brain video crowd intelligence eye-brain perception system can focus on areas of interest or relatively important semantic perception. This mechanism can be used to guide the collaborative bit rate allocation and quantitative control optimization between multiple cameras.

[0018] In the application scenario of crowd-intelligence video perception in the city brain, the distribution of video content between cameras is quite different, and the bitrate consumption of pixel-level encoded video streams of different cameras may be significantly different; in addition, some blocks or camera scenes may not have important or attention-grabbing semantic target objects for a long time, and some areas may have relatively frequent semantic targets, and the compact encoding bitrate of semantic target information of different cameras may be significantly different. From a statistical point of view, the probability of different cameras retrieving semantic targets in the cloud brain within a certain time window may also be different. The amount of information reflecting the size of the "city-level holographic perception" of semantic targets is an issue that needs to be considered when optimizing crowd-intelligence video perception encoding with limited transmission and storage resources and hierarchical transmission and storage technology architecture. Specifically, by optimizing the bitrate allocation and controlling the encoding parameters of compact bitstreams at different levels, such as quantization step size and fixed-point precision adjustment, adaptive bitrate allocation across camera content can be achieved in a large range, thereby realizing collaborative joint optimization encoding of multiple cameras.

[0019] Furthermore, in the step one, a target detection and tracking algorithm is used to perform target detection and tracking on each camera video to obtain a semantic target sequence, taking into account the semantic target detection confidence, the semantic target ambiguity in the semantic rectangular box, the semantic integrity parameter is defined based on the occlusion exposure degree and the integrity of the skeleton key points, and the average correlation of adjacent semantic target images. Based on the constraints of maximizing the retention of semantic information and maximizing the average correlation, a group of multi-angle (front, back and side) semantic target images are selected from the semantic target image sequence of each camera video.

[0020] Furthermore, in the step 2, cross-camera target tracking is performed for the semantic target image selected from the single-channel camera video, the correlation between the semantic targets is analyzed, the confidence of the semantic target detection is considered, the ambiguity of the semantic target in the semantic frame is considered, and the semantic integrity parameters are defined based on the degree of occlusion exposure and the integrity of the skeleton key points. At the same time, the time and physical space (physical position corresponding to the camera IP) constraints are considered to determine the cross-camera target trajectory sequence, generate a complete trajectory for the semantic target in the entire camera network, and utilize the physical space and time constraints and the sparsity constraints to select a group of (front, back and side) semantic target images in the cross-camera tracking trajectory based on the constraints of maximizing the retention of semantic information and maximizing the average correlation.

[0021] Furthermore, in the step three, a specific semantic target object (d0, d1, ...) is detected based on the semantic target extraction network, and feature extraction, representation and encoding driven by the dual objectives of compression and retrieval are performed. Taking into account the temporal correlation of the video signal, a target object usually stays in front of the camera for a period of time. Based on target tracking and image optimization, a semantic picture frame containing the largest semantic information is selected and encoded and described once; then the time domain trajectory is found through tracking technology, and a compact deep learning feature sequence (f0, f1, ...) and / or a more compact key point sequence feature is detected for the picture frame sequence of the target object on the time domain trajectory through feature extraction, and only the compact features are encoded into the bitstream and transmitted to the decoder, and the compact deep learning feature sequence is obtained based on the decoder decoding. And the reference object obtained by decoding Reconstructing semantic target objects using conditional generation networks

[0022] The reconstruction process is as follows:

[0023]

[0024] Among them, i represents the object category, represents the reconstructed semantic target object, G(·) represents the reconstruction function, Represents the reference object obtained by decoding, Represents a compact deep learning feature sequence obtained by decoding.

[0025] Furthermore, in the step 3, a multi-level compact feature expression method is adopted, including video group preferred semantic pictures, unstructured visual features, semantic target key point sequences, and structured visual features; for the camera containing the preferred picture, the encoding includes the preferred picture, the unstructured visual features ( Figure 2 The cross-camera semantic target ID is identified for cameras that do not contain the preferred image, and only the code stream including unstructured visual features, semantic target key point sequences and structured features is encoded to achieve efficient removal of semantic redundancy; based on rate-distortion-complexity constrained fixed-point and entropy coding, compact coding representation of cross-camera semantic targets is performed.

[0026] Furthermore, in the step four, a multi-layer feedback control link is constructed from the cloud to the edge server and then through the end-side server to the end-side camera. Based on the bit rate generated by the compact coding of each layer collected within a certain window, and the operations performed by the business system on the semantic targets transmitted at each layer, the semantic weight coefficient of each layer is measured respectively to guide the bit rate allocation of each layer.

[0027] Furthermore, the cameras associated with the cross-camera tracking target sequence are equivalently converted to obtain the camera € kji Data is transmitted through the end-side server E k Arrives at the edge server S k Compact coding produces bit rate Here, the semantic perception redundancy between cameras is compressed through collaborative coding, and the number of retrievals or accesses of the semantic targets transmitted by the corresponding server by the business system is Based on bit rate sum and Constructing semantic weight coefficient ρ k , for edge server S k Intelligent video feature coding or semantic feature coding bit rate allocation, including quantization step size and fixed-point precision adjustment, namely, end-to-end image coding quantization step size Qp and deep feature fixed-point precision.

[0028] Furthermore, the camera € kji Data is stored on the client server E k Compact coding produces bitrate The sum is The number of retrievals or accesses of semantic targets transmitted by the corresponding server by the business system is based on and The constructed semantic weight coefficient ρ kj , used for client-side server Ekj The bit rate allocation of intelligent video feature coding or semantic feature coding includes quantization step size and fixed-point precision adjustment, namely, end-to-end image coding quantization step size Qp and deep feature fixed-point precision.

[0029] Furthermore, the camera € kji Output compressed video bit rate The sum is The number of retrievals or accesses of semantic targets transmitted by the corresponding server by the business system is based on and The constructed semantic weight coefficient ρ kji , adjusts the complexity of the image content to adjust the camera€ kji Video encoding bit rate allocation to optimize camera performance kji Video encoding bit rate control, ultimately adjusting the quantization parameters to achieve bit rate control.

[0030] The third-level semantic weight coefficient ρ k , kj and ρ kji The formula is as follows:

[0031]

[0032]

[0033]

[0034] in, The function performs weighted fusion on all end-side camera networks in the coverage area of ​​the k-th edge server; The function performs weighted fusion on all camera networks in the coverage area of ​​the kj-th road-side server; The function performs weighted fusion on all cameras in the kji-th camera network coverage area.

[0035] A device-edge-cloud compact video coding system supporting crowd intelligence perception includes a cloud, an edge server, a device-side server and a camera, wherein the cloud is connected to a group of edge servers, the edge server is connected to a group of device-side servers, and the device-side server is connected to a group of cameras, and an intelligent computing power platform is deployed on the camera and / or the device-side server to perform semantic target detection and semantic target image optimization; a cross-camera semantic target optimization algorithm module is deployed on the edge server to perform cross-camera semantic target image optimization, and a semantic image frame containing the largest semantic information is selected based on target tracking and image priority through a video crowd intelligence semantic target compact coding module, and feature extraction is performed on the image frame sequence of the semantic target object on its time domain trajectory to obtain compact features for reconstruction of the semantic target object; a feedback control link is constructed from the cloud to the edge server and then to the device-side camera via the device-side server, and control parameters for feedback control are collected at different granularities for feedback adjustment, so that the city brain video crowd intelligence eye-brain perception system has the ability to focus on areas of interest or relatively important semantic perception, and this mechanism can be used to guide the video crowd intelligence multi-camera end-edge-cloud collaborative bit rate allocation and quantization control optimization.

[0036] The advantages and beneficial effects of the present invention are:

[0037] The present invention provides an end-edge-cloud compact video encoding method and system supporting crowd intelligence perception, which compactly encodes video crowd intelligence semantic targets through the optimization of single-camera and cross-camera semantic target images, and optimizes the bit rate allocation and quantization control among multiple cameras based on feedback control, thereby making full use of the cross-camera semantic redundancy, intra-frame temporal and spatial semantic redundancy and statistical redundancy characteristics of semantic target objects in multi-camera videos, and realizing multi-objective performance optimization of end-edge-cloud storage, transmission cost and coding compression efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is the framework diagram of the city brain video group intelligence perception.

[0039] Figure 2 This is the structure diagram of end-cloud collaborative computing video group intelligence.

[0040] Figure 3a It is a schematic diagram of the preferred semantic target image in a single camera of the video group space-time-camera multi-dimensional redundancy in the present invention.

[0041] Figure 3b It is a schematic diagram of the preferred cross-camera semantic picture of the video group space-time-camera multi-dimensional redundancy in the present invention.

[0042] Figure 4 This is a flow chart of human-machine collaborative perception video encoding in the present invention.

[0043] Figure 5It is a schematic diagram of the hierarchical convergence transmission and storage structure in the present invention.

[0044] Figure 6 It is a schematic diagram of the perceptual coding of group intelligence video hierarchical feedback control in the present invention.

[0045] Figure 7 It is a multi-granularity feedback control diagram of crowd-intelligence video coding in the present invention. DETAILED DESCRIPTION

[0046] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0047] In the eye-brain perception model of the urban brain, massive cameras generate extremely large amounts of video data. Feature extraction and compact coding expression of this data, achieving hierarchical and collaborative efficient storage and transmission of video data, and solving the pain point of extremely high transmission and storage costs are key issues that need to be addressed in the urban brain's video crowd intelligence perception.

[0048] The video crowd intelligence perception process of the City Brain and the visual perception process of HVS are similar in structure, such as Figure 1 As shown in the figure, the city-level big data holographic perception is abstracted into a city-level eye-brain perception system, the video big data computing architecture such as the end-side camera, edge storage computing server and cloud computing server is mapped into HVS photoreceptor cells, optic nerves and brain, and the video crowd intelligence perception process is abstracted into a collaborative perception process of city-level semantic targets with different receptive fields and different granularities, providing new ideas for the coding representation of multi-camera video crowd intelligence collaborative perception.

[0049] Using cross-camera multi-target tracking technology, Figure 2 In the end-edge cloud compact video coding system that supports crowd intelligence perception shown, a multi-target tracking algorithm module is deployed in the end-side server or edge crowd intelligence server to perform spatiotemporal trajectory analysis and spatiotemporal association on semantic targets in a camera network that is adjacent to a certain physical space. The present invention focuses on crowd intelligence perception coding technology for city brain videos to reduce the extremely high transmission, storage and computing costs faced by video big data applications. Based on the information theory source coding theory, the key to data compression and compact coding lies in how to remove the redundancy of semantic information across cameras in video crowd intelligence perception. Specifically, the end-edge cloud compact video coding method comprises the following steps:

[0050] (1) Single-camera semantic target detection / image selection: The industry's latest target detection and tracking algorithm is used to detect and track targets in each camera video to obtain a sequence of semantic target frames. Considering the confidence of semantic target detection, the ambiguity of semantic targets in the semantic rectangular frame, the semantic integrity parameters defined based on the degree of exposure and skeleton key point integrity, and the average correlation of adjacent semantic target images, the front, back, and side semantic target images are selected from the semantic target frame image sequence in each video based on the constraints of maximizing the retention of semantic information and maximizing the average correlation. These calculations are deployed on the edge intelligent server or the intelligent computing platform built into the front-end camera.

[0051] (2) Cross-camera semantic target image optimization: Deploy a cross-camera semantic target optimization algorithm module on the edge crowd intelligence computing server, perform correlation analysis on camera data in a certain range of physical space proximity, perform cross-camera target tracking on the semantic target image selected from the single-channel camera video, analyze the degree of correlation between semantic targets, and consider the semantic target detection confidence, the semantic target ambiguity in the semantic frame, and define the semantic integrity parameters based on the degree of occlusion exposure and the integrity of skeleton key points. Consider the time and physical space constraints (the physical position corresponding to the camera IP), determine the cross-camera target trajectory sequence, and generate a complete trajectory for the semantic target in the entire camera network. Using the physical space and time constraints and the sparsity constraints, the front, back, and side semantic images are selected in the cross-camera tracking trajectory based on the constraints of maximizing the retention of semantic information and maximizing the average correlation.

[0052] (3) Compact encoding of video crowd intelligence semantic targets: Crowd intelligence videos have large cross-camera semantic information redundancy, such as Figure 3a , Figure 3b As shown, in the city brain video crowd intelligence application, maximizing the compression of cross-camera semantic redundancy is the key to further reducing storage and transmission consumption. The present invention draws on the structure of 126-fold bandwidth attenuation of the HVS visual neural channel to explore an effective semantic target compact encoding method from the perspective of video crowd intelligence perception. After compact encoding and compression, the data is gathered in the cloud to realize the video crowd intelligence perception service in the cloud brain. When designing the compact expression of semantic information and the encoding scheme, compromise and balance should be made as much as possible in terms of retaining semantic information and removing semantic redundancy.

[0053] Considering the requirements of different machine vision tasks, some applications may require target object behavior analysis or sentiment analysis, such as Figure 2 As shown, the present invention adopts a multi-level compact feature expression method such as video group preferred semantic pictures, unstructured visual features, semantic target key point sequences, and structured visual features, corresponding to the HVS eye-brain perception structure. For the camera containing the preferred picture, the preferred picture, unstructured visual features ( Figure 2For cameras that do not contain the preferred image, the cross-camera semantic target ID is identified, and only the unstructured visual features, semantic target key point sequences, and structured features are encoded to achieve efficient removal of semantic redundancy. Based on the fixed-point and entropy coding technology of rate-distortion-complexity constraints, the compact coding representation of cross-camera semantic targets is achieved, as follows:

[0054] like Figure 4 As shown, based on the semantic target extraction network (SSD or YOLO, etc.), specific semantic objects (d0, d1, ...) (such as pedestrians, vehicles, faces or semantic targets of application concern) are detected, and feature extraction, representation and encoding driven by the dual goals of compression and retrieval are performed. Taking into account the temporal correlation of video signals, a target object usually stays in the camera for a period of time. Based on target tracking and image optimization, a semantic picture frame containing the largest semantic information is selected and encoded and described once; then the time domain trajectory can be found through tracking technology, and a compact deep learning feature sequence (f0, f1, ...) or even more compact key point sequence features are detected for the target object in the picture frame sequence on the time domain trajectory through feature extraction, and only the compact features are encoded into the bitstream and transmitted to the decoder; based on the decoder decoding, a compact deep learning feature sequence is obtained. And the reference object obtained by decoding Reconstructing semantic target objects using conditional generation networks The reconstruction process can be expressed as:

[0055]

[0056] Here, i represents the object category and G(·) represents the reconstruction function.

[0057] (4) Video group intelligence multi-camera end-edge-cloud collaborative optimization encoding: Figure 5 The schematic diagram of the video crowd intelligence camera network, the relationship between the edge cloud computing nodes and the bitstream convergence is given. Assume that in the edge computing architecture of the city brain, there are K edge / intermediate crowd intelligence servers S k (k=1,2,…K), each edge / intermediate server covers a camera network within a certain physical range and connects to J k End-side intelligent server E k (j=1,2,…J k ), each end-side server connects to I kj cameras, camera network marked as € kji . Webcam € kji The output data is encoded in multiple levels to obtain code stream B kji , B kj and B k .

[0058] For webcam€ kji Output compressed video bit rate; € kji The bit rate generated by further compact encoding of data on the client server; € kji The data is transmitted from the client-side server to the edge / intermediate crowd intelligence server for further compact encoding to generate bitrate. Here, the semantic perception redundancy between cameras is compressed through collaborative encoding. It is obtained by equivalent conversion of the cameras associated with the cross-camera tracking target sequence.

[0059] In the application scenario of crowd-intelligence video perception in the city brain, the camera € kji There are significant differences in the distribution of video content between different cameras, and the bitrate consumption of pixel-level encoding video streams of different cameras may be significantly different; in addition, some blocks or camera scenes may not have important or noteworthy semantic target objects for a long time, while some areas may have relatively frequent semantic targets. Figure 5 The bitrate of the compact encoding bitstream for semantic target information of different cameras may be significantly different.

[0060] From a statistical point of view, the probability of different cameras retrieving semantic targets in the cloud brain within a certain time window may also be different. The amount of information reflecting the "city-level holographic perception" of semantic targets is an issue that needs to be considered when optimizing crowd-intelligence video perception coding with limited transmission and storage resources and hierarchical transmission and storage technology architecture. Specifically, the bit rate allocation optimization and coding parameter control of compact bit streams at different levels, such as quantization step size and fixed-point precision adjustment, can be performed to achieve cross-camera content adaptive bit rate allocation in a large range, thereby achieving collaborative joint optimization coding of multiple cameras.

[0061] Inspired by the feedback perception mechanism of the HVS visual system, the present invention constructs a feedback control link from the cloud brain to the edge / intermediate-level crowd intelligence server, and then through the end-side server to the end-side camera. The control parameters used for feedback control are collected at different granularities for feedback adjustment, so that the city brain video crowd intelligence eye-brain perception system has the ability to "focus" on areas of interest or relatively important semantic perception. This mechanism can be used to guide the collaborative bit rate allocation and quantitative control optimization between multiple cameras.

[0062] like Figure 6 As shown in the figure, the cloud brain collects historical data of a certain window and uses the kth edge / intermediate group intelligence server S k For example, the physical area where J k The area covered by the end-side server is then connected to I kj cameras. Assume that the camera kjiGenerate data that flows through the cloud brain and edge / intermediate group intelligence servers S k Bit rate The sum is recorded as Among them, the camera € kji Generates data that flows through edge / intermediate crowd intelligence servers S k and client server E kj Bit rate The sum of the values ​​is recorded as Which camera € kji Generate and transmit to the end-side server E kj Bit rate The sum is recorded as In addition, within a time window, the number of retrievals or accesses (hit counts) of semantic targets transmitted by servers at the following levels in various business systems of the City Brain are respectively and

[0063] Based on the collection within a certain window and as well as and Separate measurement Figure 6 The three-level semantic weight coefficient ρ is shown k , kj and ρ kji as follows:

[0064]

[0065] in, The function performs weighted fusion on all end-side camera networks in the coverage area of ​​the k-th edge / intermediate crowd intelligence server; The function performs weighted fusion on all camera networks in the coverage area of ​​the kj-th road-side server; The function performs weighted fusion on all cameras in the kji-th camera network coverage area.

[0066] like Figure 7 As shown, the semantic weight coefficient ρ k and ρ kj , which are used to guide the edge / intermediate level crowd intelligence server S k and client server E kj Intelligent video feature coding or semantic feature coding bit rate allocation, including quantization step size, fixed-point precision adjustment, i.e., end-to-end image coding quantization step size Qp and deep feature fixed-point precision; the semantic weight coefficient ρ kji , used to optimize the camera€ kjiVideo encoding bit rate control, specifically, when measuring the complexity of image content in the bit rate allocation link, adjusts the complexity through the semantic weight coefficient to achieve bit rate allocation fine-tuning, and finally adjusts the quantization parameter to achieve bit rate control.

[0067] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for edge-cloud compact video encoding supporting crowd intelligence perception, characterized in that The steps include: Step 1: Single camera semantic target detection and semantic target image selection; Step 2: cross-camera semantic target image optimization: for the semantic target images of the cameras adjacent to each other in a certain range of physical space, a cross-camera semantic target optimization algorithm is used to perform association analysis, generate a cross-camera semantic target track, and select a group of semantic target images from the semantic target track; Step 3: Compact encoding of video crowd intelligence semantic targets; Based on target tracking and image priority, the semantic image frame containing the largest semantic information is selected, and the feature extraction of the image frame sequence of the semantic target object on its time domain trajectory is performed, and the obtained compact features are used to reconstruct the semantic target object; The semantic target object is detected based on the semantic target extraction network. After feature extraction, representation and encoding driven by the dual goals of compression and retrieval, the semantic picture frame containing the largest semantic information is selected based on target tracking and picture optimization, and an encoding description is performed on it. Then, the time domain trajectory is found through tracking technology, and a compact deep learning feature sequence and / or a more compact key point sequence feature is detected for the picture frame sequence of the target object on the time domain trajectory through feature extraction. Only the compact features are encoded into the bitstream and transmitted to the decoder. Based on the compact deep learning feature sequence obtained by decoding by the decoder and the reference object obtained by decoding, the semantic target object is reconstructed using a conditional generation network. Step 4: Video group intelligent multi-camera collaborative optimization encoding; Collect control parameters for feedback control, focus on areas of interest or relatively important semantic perception, and guide the coordinated bit rate allocation and quantization control optimization among multiple cameras; Build a multi-layer feedback control link from the cloud to the edge server and then to the end-side camera via the end-side server. Based on the bit rate generated by the compact coding of each layer collected within a certain window and the operations performed by the business system on the semantic targets transmitted at each layer, measure the semantic weight coefficient of each layer to guide the bit rate allocation of each layer. The cameras associated with the cross-camera tracking target sequence are equivalently converted to obtain the camera € kji Data is transmitted through the end-side server E k Arrives at the edge server S k Compact coding produces bit rate The number of semantic targets transmitted by the corresponding server that are retrieved or accessed by the business system is Based on bitrate sum and Constructing semantic weight coefficient ρ k , for edge server S k Bit rate allocation for intelligent video feature coding or semantic feature coding; Webcam€ kji Data is stored on the client server E k Compact coding produces bitrate The sum is The number of semantic targets transmitted by the corresponding server that are retrieved or accessed by the business system is based on and The constructed semantic weight coefficient ρ kj , used for client-side server E kj Bit rate allocation for intelligent video feature coding or semantic feature coding; Webcam€ kji Output compressed video bit rate The sum is The number of semantic targets transmitted by the corresponding server that are retrieved or accessed by the business system is based on and The constructed semantic weight coefficient ρ kji , adjusts the complexity of the image content to adjust the camera€ kji Bit rate allocation for video encoding; Based on the collection within a certain window and as well as and Measure the three-level semantic weight coefficients ρ respectively k , kj and ρ kji The formula is as follows: in, The function performs weighted fusion on all end-side camera networks in the coverage area of ​​the k-th edge / intermediate crowd intelligence server; The function performs weighted fusion on all camera networks in the coverage area of ​​the kj-th road-side server; The function performs weighted fusion on all cameras in the network coverage area of ​​the kji-th camera; The obtained semantic weight coefficient ρ k and ρ kj , which are used to guide the edge / intermediate level crowd intelligence server S k and client server E kj Intelligent video feature coding or semantic feature coding bit rate allocation, including quantization step size, fixed-point precision adjustment, i.e., end-to-end image coding quantization step size Qp and deep feature fixed-point precision; the semantic weight coefficient ρ kji , used to optimize the camera€ kji Video encoding bit rate control, specifically, when measuring the complexity of image content in the bit rate allocation link, adjusts the complexity through the semantic weight coefficient to achieve bit rate allocation fine-tuning, and finally adjusts the quantization parameter to achieve bit rate control.

2. According to claim 1, a method for edge-cloud compact video encoding supporting crowd intelligence perception is characterized in that: In the step 1, a target detection and tracking algorithm is used to perform target detection and tracking on each camera video to obtain a semantic target sequence, and the semantic target detection confidence, the semantic target ambiguity in the semantic rectangular box, the semantic integrity parameter is defined based on the occlusion exposure degree and the skeleton key point integrity, and the average correlation of adjacent semantic target images is considered. Based on the constraints of maximizing the retention of semantic information and maximizing the average correlation, a group of multi-angle semantic target images are selected from the semantic target image sequence of each camera video.

3. The method for edge-cloud compact video encoding supporting crowd intelligence perception according to claim 1, characterized in that: In the step 2, cross-camera target tracking is performed for the semantic target image selected from the single-channel camera video, the correlation between the semantic targets is analyzed, the confidence of the semantic target detection is considered, the ambiguity of the semantic target in the semantic frame is considered, and the semantic integrity parameters are defined based on the occlusion exposure degree and the integrity of the skeleton key points. The time and physical space constraints are considered at the same time, and the cross-camera target trajectory sequence is determined to generate a complete trajectory for the semantic target in the entire camera network. The physical space and time constraints and the sparsity constraints are used to select a group of semantic target images in the cross-camera tracking trajectory based on the constraints of maximizing the retention of semantic information and maximizing the average correlation.

4. The method for edge-cloud compact video encoding supporting crowd intelligence perception according to claim 1, characterized in that: In the step 3, a multi-level compact feature expression method is adopted, including video group preferred semantic pictures, unstructured visual features, semantic target key point sequences, and structured visual features; For cameras that contain preferred images, the code stream including the preferred images, unstructured visual features, semantic target key point sequences and structured features is encoded; for cameras that do not contain preferred images, the cross-camera semantic target ID is identified, and only the code stream including unstructured visual features, semantic target key point sequences and structured features is encoded; based on rate-distortion-complexity constrained fixed-point and entropy coding, compact coding representation of cross-camera semantic targets is performed.

5. A system supporting the end-edge-cloud compact video encoding method supporting crowd intelligence perception as claimed in claim 1, comprising a cloud, an edge server, an end-side server and a camera, the cloud is connected to a group of edge servers, the edge server is connected to a group of end-side servers, and the end-side server is connected to a group of cameras, characterized in that: An intelligent computing platform is deployed on the camera and / or the end-side server to perform semantic target detection and semantic target image optimization; a cross-camera semantic target optimization algorithm module is deployed on the edge server to perform cross-camera semantic target image optimization, and a semantic image frame containing the largest semantic information is selected based on target tracking and image priority through a video group intelligence semantic target compact encoding module, and feature extraction is performed on the image frame sequence of the semantic target object on its time domain trajectory to obtain compact features for reconstruction of the semantic target object; a feedback control link is constructed from the cloud to the edge server and then to the end-side camera via the end-side server, and control parameters for feedback control are collected at different granularities, focusing on areas of interest or relatively important semantic perception, to guide the video group intelligence multi-camera end-edge-cloud collaborative bitrate allocation and quantization control optimization; The cameras associated with the cross-camera tracking target sequence are equivalently converted to obtain the camera € kji Data is transmitted through the end-side server E k Arrives at the edge server S k Compact coding produces bit rate The number of semantic targets transmitted by the corresponding server that are retrieved or accessed by the business system is Based on bitrate sum and Constructing semantic weight coefficient ρ k , for edge server S k Bit rate allocation for intelligent video feature coding or semantic feature coding; Webcam€ kji Data is stored on the client server E k Compact coding produces bitrate The sum is The number of semantic targets transmitted by the corresponding server that are retrieved or accessed by the business system is based on and The constructed semantic weight coefficient ρ kj , used for client-side server E kj Bit rate allocation for intelligent video feature coding or semantic feature coding; Webcam€ kji Output compressed video bit rate The sum is The number of semantic targets transmitted by the corresponding server that are retrieved or accessed by the business system is based on and The constructed semantic weight coefficient ρ kji , adjusts the complexity of the image content to adjust the camera€ kji Bit rate allocation for video encoding; Based on the collection within a certain window and as well as and Measure the three-level semantic weight coefficients ρ respectively k , kj and ρ kji The formula is as follows: in, The function performs weighted fusion on all end-side camera networks in the coverage area of ​​the k-th edge / intermediate crowd intelligence server; The function performs weighted fusion on all camera networks in the coverage area of ​​the kj-th road-side server; The function performs weighted fusion on all cameras in the network coverage area of ​​the kji-th camera; The obtained semantic weight coefficient ρ k and ρ kj , which are used to guide the edge / intermediate level crowd intelligence server S k and client server E kj Intelligent video feature coding or semantic feature coding bit rate allocation, including quantization step size, fixed-point precision adjustment, i.e., end-to-end image coding quantization step size Qp and deep feature fixed-point precision; the semantic weight coefficient ρ kji , used to optimize the camera€ kji Video encoding bit rate control, specifically, when measuring the complexity of image content in the bit rate allocation link, adjusts the complexity through the semantic weight coefficient to achieve bit rate allocation fine-tuning, and finally adjusts the quantization parameter to achieve bit rate control.

Images