A deep learning-based image lightweight encoding method and system

Abstract

The application discloses a kind of based on deep learning's image light weight encoding method and system, belong to deep learning technical field.The application uses convolutional neural network to train deep learning model, identifies current comprehensive environment;Generator and discriminator are constructed, and using the training of opposition, generator and discriminator are alternately trained, the encoding optimization of current security monitoring image is carried out, and simplified image is generated;Simplified image is input into image encoding algorithm and is encoded, and first light weight encoding image is obtained;Multi-source data features are extracted, and multi-source data feature templates under different comprehensive environments are established, comparative analysis is carried out, and the matching degree of environment and template is obtained;The information retention of target area in first light weight encoding image after light weight encoding is determined;Based on information retention, first light weight encoding image is adjusted, and after adjustment, it is encoded according to selected image encoding algorithm, and final light weight encoding image is generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to an image lightweight coding method and system based on deep learning. BACKGROUND

[0002] In the process of smart city construction, urban street security monitoring, as an important line of defense to protect residents' safety and maintain social order, is becoming increasingly important. Urban streets are crisscrossed, with frequent personnel and vehicle traffic, and complex and variable public security and traffic conditions. In order to comprehensively and real-time grasp the street dynamics, a large number of high-definition cameras are deployed at various intersections, road sections and key public places. These cameras collect massive image data day and night, which can be used for security monitoring.

[0003] During the peak period of urban streets, the number of monitoring images generated per unit time is huge. Traditional image coding algorithms have high computational complexity and cumbersome processing procedures, and the coding speed is far from keeping up with the data acquisition speed. Urban street monitoring data needs to be stored for a long time for subsequent analysis and verification. The image data files generated by the traditional unoptimized coding method have a large file size and occupy a large amount of storage space. The urban street environment is complex and diverse, facing low light, bad weather and complex light, etc. Traditional coding techniques lack effective adaptability mechanisms when dealing with these complex environments. SUMMARY

[0004] The purpose of the present application is to provide an image lightweight coding method and system based on deep learning to solve the problems in the prior art.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides an image lightweight coding method based on deep learning, comprising:

[0007] Collecting historical urban security monitoring multi-source data, training a deep learning model using a convolutional neural network, performing model training, evaluation and optimization, and identifying the current comprehensive environment;

[0008] Based on the current comprehensive environment, a generator and a discriminator are constructed, and the generator and the discriminator are alternately trained using adversarial training to optimize the coding of the current security monitoring image and generate a simplified image. The simplified image is input into an image coding algorithm for coding to obtain a first lightweight coded image;

[0009] Collecting current urban security monitoring multi-source data, extracting multi-source data features; based on the current comprehensive environment, establishing multi-source data feature templates under different comprehensive environments, comparing and analyzing the multi-source data features and the corresponding multi-source data feature templates, and obtaining the matching degree of the environment and the template;

[0010] determine the information retention of the target region in the first lightweight encoded image after lightweight encoding based on the matching degree;

[0011] adjust the first lightweight encoded image based on the information retention, and encode the adjusted image according to the selected image encoding algorithm to generate a final lightweight encoded image.

[0012] In combination with the first aspect, in a first implementation manner of the first aspect of the present application, the historical urban security monitoring multi-source data is collected, a deep learning model is trained using a convolutional neural network, model training, evaluation and optimization are performed, and the current comprehensive environment is identified, including:

[0013] Images under different weathers, different time periods and different scenes are collected from urban street cameras, laser radar data and millimeter wave radar data are synchronously collected, the laser radar data provide three-dimensional point cloud information of the environment around the street, the millimeter wave radar data include distance, speed and angle information of target objects, and the historical urban security monitoring multi-source data is integrated;

[0014] The historical urban security monitoring multi-source data is preprocessed and divided into a training set, a validation set and a test set, a ResNet is selected as the structure of the convolutional neural network, a loss function, an optimizer and a number of training rounds are set, image data and corresponding sensor data in the training set are input into the deep learning model, the model calculates a prediction result through forward propagation, a loss value is calculated using the loss function according to the difference between the prediction result and a real label, a gradient of the loss value to model parameters is calculated using a back propagation algorithm, and the model parameters are updated using the optimizer, so that the prediction result of the model gradually approaches the real label, the performance of the model is evaluated on the validation set after each round of training is completed, and the loss value and the accuracy on the validation set are recorded, and model evaluation and optimization are performed;

[0015] Image data and sensor data around the urban street are collected in real time, preprocessed and input into the deep learning model, and the model outputs the current comprehensive environment.

[0016] In combination with the first aspect, in a second implementation manner of the first aspect of the present application, based on the current comprehensive environment, a generator and a discriminator are constructed, the generator and the discriminator are alternately trained using adversarial training, the current security monitoring image is encoded and optimized, and a simplified image is generated, including:

[0017] An encoder-decoder structure is adopted as a basic framework of the generator, and image features of the current security monitoring image are extracted by the encoder; an environment category of the current comprehensive environment is coded into a one-hot vector, and after being spliced with the image features extracted by the encoder, the one-hot vector is input into the generator as a condition; a skip connection is added between the encoder and the decoder of the generator, and features at different levels in the encoder are connected to corresponding levels of the decoder; the generator generates a to-be-detected simplified image;

[0018] A discriminator based on a convolutional neural network is constructed, an input of the discriminator is the to-be-detected simplified image and the current security monitoring image, the discriminator is composed of multiple convolutional layers, and features of the image are extracted by convolutional operation; in the discriminator, multi-scale feature fusion is adopted to discriminate the authenticity and quality of the image; an output layer of the discriminator is a binary classification layer, a sigmoid activation function is used, and a value between 0 and 1 is output, indicating a probability that the to-be-detected simplified image is the current security monitoring image;

[0019] A training set is prepared, the generator and the discriminator are initialized, and the generator and the discriminator are trained in an adversarial manner to generate a simplified image.

[0020] In combination with the first aspect, in a third implementation manner of the first aspect of the present application, the preparation of the training set, the initialization of the generator and the discriminator, the adversarial training, and the generation of the simplified image include:

[0021] The training set uses historical urban security monitoring multi-source data to initialize weight parameters of the generator and the discriminator, and the generator and the discriminator are compiled in a TensorFlow framework; and the number of adversarial training rounds of the generator and the discriminator is determined.

[0022] In the training process, the loss value change of the generator and the discriminator is monitored in real time to determine whether the generator and the discriminator converge; the current security monitoring image and the corresponding current comprehensive environment are input into the generator, and the generator outputs a simplified image.

[0023] In combination with the first aspect, in a fourth implementation manner of the first aspect of the present application, the input of the simplified image into the image encoding algorithm for encoding to obtain a first lightweight encoded image includes:

[0024] The user specifies an image encoding algorithm, sets an encoding parameter, and performs data format conversion including color space conversion and image size adaptation; the simplified image is encoded according to an encoding process of the different encoding algorithms, and MD5 verification is performed after encoding to obtain a first lightweight encoded image.

[0025] In combination with the first aspect, in a fifth implementation manner of the first aspect of the present application, the collection of the current urban security monitoring multi-source data and the extraction of multi-source data features include:

[0026] Collect current urban security monitoring multi-source data, including current security monitoring image, current laser radar data and current millimeter wave radar data; extract current security monitoring image features, including color features, texture features and shape features, extract current laser radar data features, including geometric features and motion features, the geometric features include point cloud density features, object geometric shape features and surface normal vector features, extract current millimeter wave radar data features, including target object attribute features and target object motion mode features, the target object attribute features include distance features, velocity features and angle features.

[0027] In combination with the first aspect, in a sixth implementation manner of the first aspect of the present application, based on the current comprehensive environment, a multi-source data feature template under different comprehensive environments is established, the multi-source data features are compared and analyzed with the corresponding multi-source data feature template, and a matching degree of the environment and the template is obtained, including:

[0028] Based on the environment category of the comprehensive environment, statistical analysis is performed to obtain multi-source data feature templates, including security monitoring image feature templates, laser radar data feature templates and millimeter wave radar data feature templates; the current security monitoring image features, the current laser radar data features and the current millimeter wave radar data features in the multi-source data features are respectively compared with the security monitoring image feature templates, the laser radar data feature templates and the millimeter wave radar data feature templates, the comprehensive similarity score is calculated by setting the weights of different comparison results; and the matching degree of the current environment and the template is determined according to the comprehensive similarity score.

[0029] In combination with the first aspect, in a seventh implementation manner of the first aspect of the present application, based on the matching degree, the information retention situation of the target region in the first lightweight coded image after lightweight coding is determined, including:

[0030] When the matching degree of the current environment and the template is higher than the set threshold, according to the target region feature in the established multi-source data feature template, a DBSCAN algorithm is used in the point cloud data in the current laser radar data, and by setting the neighborhood radius and the minimum point number parameters, the target region is located; in combination with the target position provided by the millimeter wave radar, by using the coordinate conversion and matching algorithm, the target position detected by the millimeter wave radar is associated with the target region in the point cloud data, and the accuracy of the target region is confirmed; for the located target region, the accuracy thereof is verified through multi-source data cross verification; according to the target region contour information in the multi-source data feature template, the outline retention of the target region in the first lightweight coded image is evaluated, and the specific method is as follows: key points are extracted on the outline of the target region in the template and the outline of the first lightweight coded image, the Euclidean distance between the key points is calculated, the outline similarity is obtained, and when the outline similarity is higher than the set threshold, it is judged that the outline retention is normal; for the color feature and the shape feature of the target object, quantitative evaluation is performed, and the specific method is as follows: for the color feature, the color histogram is used to compare the color of the target object in the first lightweight coded image with the color in the template; for the shape feature, a shape matching algorithm based on Hu moment is used to calculate the similarity of the shape of the target object in the first lightweight coded image and the shape in the template.

[0031] When the matching degree of the current environment and the template is lower than the set threshold, the difference between the multi-source data feature and the multi-source data feature template is analyzed to obtain an abnormal data feature; according to the abnormal data feature, a Canny algorithm is used in the first lightweight coded image, and in combination with morphological processing, an object outline is found; in combination with the laser radar data and the millimeter wave radar data, a data fusion algorithm based on Kalman filtering is used to determine the position and range of the target region; a topological structure matching algorithm based on the outline shape is used to calculate the outline retention; by using a data fusion algorithm based on a Bayesian network, the current urban security monitoring multi-source data is fused, and according to the probability relationship between the current urban security monitoring multi-source data, the information retention of the target region is judged.

[0032] In combination with the first aspect, in an eighth implementation manner of the first aspect of the present application, the first lightweight coded image is adjusted based on the information retention, and after the adjustment, the first lightweight coded image is encoded according to the selected image encoding algorithm to generate a final lightweight coded image, and the specific method is as follows:

[0033] Based on the information retention, contour repair, color adjustment, shape adjustment and texture adjustment are performed; wherein, the contour repair uses the image repair algorithm based on PatchMatch, the color adjustment uses the method based on histogram matching, the shape adjustment uses the affine transformation method, and the texture adjustment uses the sample-based texture synthesis algorithm; according to the characteristics of the adjusted image, the parameters are optimized, the adjusted image is encoded according to the selected image encoding algorithm, and the final lightweight encoded image is generated.

[0034] In a second aspect, the present application provides a deep learning-based image lightweight encoding system, comprising:

[0035] The comprehensive environment recognition module comprises a deep learning model training unit and a comprehensive environment recognition unit; wherein the deep learning model training unit collects historical city security monitoring multi-source data, trains a deep learning model using a convolutional neural network, and performs model training, evaluation and optimization, and the comprehensive environment recognition unit recognizes the current comprehensive environment;

[0036] The lightweight encoded image transfer module comprises a simplified image generation unit and a first lightweight encoded image generation unit; wherein the simplified image generation unit constructs a generator and a discriminator based on the current comprehensive environment, adopts adversarial training, alternately trains the generator and the discriminator, performs encoding optimization on the current security monitoring image, and generates a simplified image; the first lightweight encoded image generation unit inputs the simplified image into an image encoding algorithm for encoding to obtain a first lightweight encoded image;

[0037] The matching degree calculation module comprises a multi-source data feature extraction unit, a feature template establishment unit and a matching degree calculation unit; wherein the multi-source data feature extraction unit collects current city security monitoring multi-source data and extracts multi-source data features; the feature template establishment unit establishes multi-source data feature templates under different comprehensive environments based on the current comprehensive environment, compares and analyzes the multi-source data features with the corresponding multi-source data feature templates, and obtains the matching degree of the environment and the template;

[0038] The information retention condition calculation module comprises an information retention condition calculation unit; wherein the information retention condition calculation unit determines the information retention condition of the target region in the first lightweight encoded image after lightweight encoding based on the matching degree;

[0039] The lightweight encoded image generation module comprises a lightweight encoded image adjustment unit and a lightweight encoded image generation unit; wherein the lightweight encoded image adjustment unit adjusts the first lightweight encoded image based on the information retention condition, and after adjustment, the lightweight encoded image generation unit encodes according to the selected image encoding algorithm to generate the final lightweight encoded image.

[0040] Compared with the prior art, the present application has the beneficial effects that:

[0041] 1、The present application uses a deep learning model to accurately identify the current comprehensive environment of urban streets, constructs a generator and a discriminator according to different environmental characteristics, and uses an adversarial training method to encode and optimize the monitoring image; during the busy period of urban street traffic, the image content in the complex scene can be quickly analyzed, the encoding parameters and the process can be intelligently adjusted, and a simplified image can be quickly generated.

[0042] 2、The present application collects multi-source data under different weather, time periods and scenes of urban streets, and establishes a multi-source data feature template; in the actual monitoring process, the encoding strategy is flexibly adjusted according to the matching degree of the current environment and the template.

[0043] 3、The light-weight encoded image generated by the present application greatly reduces the data amount on the premise of ensuring the integrity of the key information of the image; in the storage of urban street monitoring data, the storage space requirement can be significantly reduced, and a large amount of storage cost can be saved. BRIEF DESCRIPTION OF DRAWINGS

[0044] Fig. 1 is a step schematic diagram of an image light-weight encoding method based on deep learning of the present application;

[0045] Fig. 2 is a system structure diagram of an image light-weight encoding system based on deep learning of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0047] Embodiment: As shown in the accompanying drawings, Figs. 1-2 the present application provides a technical solution,

[0048] As shown in the accompanying drawings, Fig. 1 a step schematic diagram of an image light-weight encoding method based on deep learning, the present application provides an image light-weight encoding method based on deep learning, which comprises:

[0049] Step S100: Collect historical urban security monitoring multi-source data, use a convolutional neural network to train a deep learning model, perform model training, evaluation and optimization, and identify the current comprehensive environment;

[0050] Specifically, images under different weather, different time periods and different scenes are collected from urban street cameras, and laser radar data and millimeter wave radar data are collected synchronously; the laser radar data provides three-dimensional point cloud information of the environment around the street, and the millimeter wave radar data includes distance, speed and angle information of the target object; historical urban security monitoring multi-source data is formed by integration;

[0051] The historical urban security monitoring multi-source data is preprocessed and divided into a training set, a validation set and a test set; the convolutional neural network structure is selected as ResNet, the loss function, the optimizer and the number of training rounds are set; the image data and the corresponding sensor data in the training set are input into the deep learning model, the model calculates the prediction result through forward propagation, and the loss value is calculated using the loss function according to the difference between the prediction result and the true label; the gradient of the loss value to the model parameters is calculated by the back propagation algorithm, and the model parameters are updated using the optimizer, so that the prediction result of the model gradually approaches the true label; after each round of training is completed, the performance of the model is evaluated on the validation set, and the loss value and the accuracy on the validation set are recorded; model evaluation and optimization are performed;

[0052] Real-time image data and sensor data around urban streets are collected, preprocessed and input into a deep learning model, and the model outputs the current comprehensive environment.

[0053] In a specific embodiment, in the core area of a medium-sized city, 50 representative urban street monitoring cameras are selected, and data is continuously collected for 3 months. 10 laser radar devices are deployed at key intersections and road sections, and image data is collected synchronously. 20 millimeter wave radar sensors are installed in some monitoring areas, and the working frequency band is 77GHz.

[0054] Data preprocessing is performed, and the preprocessed multi-source data is divided into a training set, a validation set and a test set according to the proportions of 70%, 15% and 15%. The training set contains about 630,000 samples, and the validation set and the test set each contain about 135,000 samples. ResNet-101 is selected as the convolutional neural network structure. The loss function adopts the cross-entropy loss function, the optimizer selects Adam, the learning rate is set to 0.0001, beta1=0.9, beta2=0.999. The number of training rounds is set to 150 rounds.

[0055] During the training process, each round of training inputs the image data and corresponding lidar and millimeter wave radar data in the training set into the deep learning model. The model calculates the prediction results through forward propagation, for example, at the 30th round of training, for a batch of training data containing 128 samples, the model predicts the comprehensive environment category, and the cross-entropy loss value is calculated by comparing it with the true label. The loss value gradient is calculated for the model parameters through the back propagation algorithm, and the model parameters are updated using the Adam optimizer. During the training process, the loss value and accuracy on the validation set are recorded for each round of training. As the number of training rounds increases, the loss value on the validation set gradually decreases, and the accuracy gradually increases. At the 80th round of training, the loss value on the validation set decreases to about 0.4, and the accuracy reaches 85%. By the end of the 150th round of training, the loss value on the validation set stabilizes at about 0.25, and the accuracy reaches 92%. The trained model is evaluated using the test set. In the test set, there are samples of various weather, time period, and scene combinations. The model predicts 135,000 samples in the test set, and finally gets an accuracy of 90%, a recall rate of 88%, and an F1 value of 89%. Individual evaluation is performed on different environment categories. In the sunny main road scene, the model's accuracy is as high as 96%, because the data features of this scene are relatively obvious, and the model is easy to learn and recognize. In the rainy night secondary road scene, the accuracy is relatively low, at 82%, mainly because the light is dark and the rain is disturbing in this scene, and the data features are complex, increasing the difficulty of model recognition.

[0056] In the actual monitoring process of urban streets, image data and sensor data around vehicles are collected in real time. For example, a set of image data and sensor data is collected in real time in the evening of a working day, and after preprocessing, it is input into the deep learning model. The model can quickly output the current comprehensive environment category. In the continuous 1-hour real-time monitoring process, a total of 3600 environment recognitions are performed, of which 3200 are accurately recognized, and the accuracy rate reaches 88.9%, basically meeting the real-time and accuracy requirements of urban street security monitoring for environment recognition.

[0057] Step S200: based on the current comprehensive environment, construct the generator and the discriminator, adopt the adversarial training, alternately train the generator and the discriminator, perform the encoding optimization of the current security monitoring image, and generate a simplified image; input the simplified image into the image encoding algorithm for encoding to obtain a first lightweight encoded image;

[0058] Specifically, an encoder-decoder structure is used as the basic framework of the generator, and the image features of the current security monitoring image are extracted through the encoder; the environment category of the current comprehensive environment is encoded into a one-hot vector, which is spliced with the image features extracted by the encoder and then input into the generator as a condition; a skip connection is added between the encoder and the decoder of the generator to connect the features of different levels in the encoder to the corresponding levels of the decoder; the generator generates the to-be-detected simplified image.

[0059] A discriminator based on a convolutional neural network is constructed, the input of the discriminator is the to-be-detected simplified image and the current security monitoring image, the discriminator is composed of multiple convolutional layers, and the features of the image are extracted through convolution operation; the authenticity and quality of the image are discriminated by using multi-scale feature fusion in the discriminator; the output layer of the discriminator is a binary classification layer, which uses a sigmoid activation function and outputs a value between 0 and 1, representing the probability that the to-be-detected simplified image is the current security monitoring image.

[0060] Prepare the training set, initialize the generator and the discriminator, and perform adversarial training to generate the simplified image.

[0061] Further, the training set uses historical urban security monitoring multi-source data to initialize the weight parameters of the generator and the discriminator, and the generator and the discriminator are compiled in the TensorFlow framework; the number of adversarial training rounds of the generator and the discriminator is determined.

[0062] During the training process, the loss value changes of the generator and the discriminator are monitored in real time to determine whether they converge; the current security monitoring image and the corresponding current comprehensive environment are input into the generator, and the generator outputs the simplified image.

[0063] Further, the user specifies the image encoding algorithm, sets the encoding parameters, and performs data format conversion, including color space conversion and image size adaptation; the simplified image is encoded according to the encoding process of different encoding algorithms, and MD5 verification is performed after encoding to obtain the first lightweight encoded image.

[0064] In a specific embodiment, based on the encoder-decoder structure, the encoder part adopts a series of 3x3 convolution kernel, convolution layer with a step of 2. The input of the current security monitoring image of 224x224 pixels, after the first convolution layer, the image size becomes 112x112 pixels, and the channel number increases from 3 to 64. With the increase of the number of layers, the image is finally compressed into a low-dimensional feature representation. The decoder uses a 3x3 deconvolution kernel, a deconvolution layer with a step of 2, and gradually up-samples the low-dimensional features back to the original image size. A skip connection is added between the encoder and the decoder, for example, the features of the second layer of the encoder are spliced with the input of the corresponding position deconvolution layer of the decoder. If there are 8 categories in the current comprehensive environment, they are encoded into an 8-dimensional one-hot vector, which is spliced with the features extracted by the encoder in the channel dimension and then input into the generator. The discriminator is composed of multiple convolution layers, and the convolution kernel size is 3x3 or 5x5, and the step is set to 1 or 2. The input is the simplified image to be detected and the current security monitoring image, which is compressed into a fixed size vector after multiple convolution operations. Multi-scale feature fusion is adopted, and the input image is down-sampled by 1 / 2, 1 / 4 and 1 / 8, and the features are spliced after being extracted respectively. The output layer is a binary classification layer, which uses a sigmoid activation function to output a value between 0 and 1.

[0065] The image data in the previously collected historical urban security monitoring multi-source data is used as the training set, and 10,000 images are selected, at least 50 for each weather, time period, and scene combination. The weight parameters of the generator and the discriminator are initialized with a Gaussian distribution (mean 0, standard deviation 0.02), and the bias parameters are initialized to 0. In the TensorFlow framework, both the generator and the discriminator use the binary cross-entropy loss function, the optimizer selects Adam, the generator learning rate is set to 0.0002, the discriminator learning rate is set to 0.0001, and beta1 is set to 0.5. The number of rounds of adversarial training is set to 300 rounds. The user specifies the JPEG encoding algorithm. The quality factor is set to 70, and the quantization table uses the default standard table. For color space conversion, the RGB image is converted to YUV color space. In terms of image size adaptation, the simplified image size is ensured to be 224x224 pixels, and if the size does not match, it is cropped or padded.

[0066] The model is trained, and as the number of training rounds increases, the generator and discriminator loss values gradually decrease and tend to be stable. At the 200th training round, the generator loss value decreases to about 0.4, and the discriminator loss value decreases to about 0.35, basically reaching a convergent state. After training is completed, the new current security monitoring image and the corresponding comprehensive environment are input into the generator. In actual testing, 100 current security monitoring images under different environments are processed, and the generator successfully outputs simplified images. From the visual effect, in the sunny intersection scene, the simplified image clearly retains the key information of traffic lights, vehicles, pedestrians, etc., and only removes part of the background detail texture; in the rainy night secondary road scene, although the overall clarity of the image has decreased, the road profile, roadside buildings, and moving pedestrians and vehicles are still distinguishable.

[0067] The generated simplified image is JPEG encoded. After encoding, MD5 verification is performed to ensure data integrity.

[0068] Step S300: Collecting current urban security monitoring multi-source data, extracting multi-source data features; based on the current comprehensive environment, establishing multi-source data feature templates under different comprehensive environments, comparing and analyzing the multi-source data features and the corresponding multi-source data feature templates to obtain the matching degree of the environment and the template;

[0069] Specifically, the current urban security monitoring multi-source data is collected, including the current security monitoring image, the current laser radar data and the current millimeter wave radar data; the current security monitoring image features are extracted, including color features, texture features and shape features, the current laser radar data features are extracted, including geometric features and motion features, the geometric features include point cloud density features, object geometric shape features and surface normal vector features, the current millimeter wave radar data features are extracted, including target object attribute features and target object motion mode features, the target object attribute features include distance features, speed features and angle features.

[0070] Further, based on the environment category of the comprehensive environment, statistical analysis is performed to obtain multi-source data feature templates, including security monitoring image feature templates, laser radar data feature templates and millimeter wave radar data feature templates; the current security monitoring image features, the current laser radar data features and the current millimeter wave radar data features in the multi-source data features are compared with the security monitoring image feature templates, the laser radar data feature templates and the millimeter wave radar data feature templates respectively, and the comprehensive similarity score is calculated by setting the weights of different comparison results; according to the comprehensive similarity score, the matching degree of the current environment and the template is determined.

[0071] In a specific embodiment, statistical analysis is performed on the color, texture and shape features of 250 images under each environment, and the mean, median, standard deviation and other statistical quantities of each feature are calculated to construct the feature template. For example, in the sunny highway scene, in the image color feature template, the blue tone of the sky area in the RGB space has a mean of [100, 149, 237] and a variance of [10, 12, 15]; in the texture feature template, the GLCM energy of the road area has a mean of 0.12, an entropy mean of 3.2, and a contrast mean of 0.35; in the shape feature template, the vehicle contour Hu moment has a mean of [0.001, 0.002, 0.0015, -0.0005, 0.0003, -0.0002, 0.0001].

[0072] The laser radar data features under each environment are statistically analyzed. For example, in the rainy urban street scene, in the point cloud density feature template, the point cloud density of the building area has a mean of 200 points per cubic meter and a standard deviation of 50 points per cubic meter; in the object geometry feature template, the length of the vehicle has a mean of 4.5 meters and a width of 1.8 meters; in the surface normal vector feature template, the main direction of the building wall surface normal vector has a mean of [0, 0, 1] (vertical direction); in the motion feature template, the average speed of the vehicle in this environment is 25 km / h, and the average acceleration is 0.5 m / s2.

[0073] The millimeter wave radar data features under each environment are statistically analyzed. In the night rural road scene, in the target object attribute feature template, the distance feature has a mean of 80 meters, the speed feature has a mean of 30 km / h, and the angle feature has a mean of 0° (straight ahead); in the target object motion mode feature template, the probability of uniform linear motion is 0.7, and the probability of turning motion is 0.2.

[0074] The weights of the comparison results of the current security monitoring image features, the current laser radar data features and the current millimeter wave radar data features are set to 0.4, 0.3 and 0.3 respectively. The Euclidean distance, cosine similarity and other methods are used to calculate the similarity between the current features and the template features. For color features, the Euclidean distance of the color histogram is calculated; for texture features, the cosine similarity of GLCM parameters and LBP pattern distribution is calculated; for shape features, the Euclidean distance of Hu moment is calculated. For laser radar and millimeter wave radar data features, the corresponding distance or similarity measurement method is also used.

[0075] Randomly select 50 current security monitoring images and their corresponding lidar and millimeter wave radar data in each environment for feature comparison. For example, in the cloudy urban arterial scene, the Euclidean distance between the color features of a current security monitoring image and the template features is calculated to obtain a similarity of 0.8, the texture feature cosine similarity is 0.75, and the shape feature Euclidean distance similarity is 0.85. Compared with the template, the corresponding lidar data features have a geometric feature similarity of 0.7 and a motion feature similarity of 0.72. Compared with the template, the millimeter wave radar data features have a target object attribute feature similarity of 0.8 and a target object motion pattern feature similarity of 0.78. According to the weight calculation, the comprehensive similarity score is: (0.8x0.4+0.75x0.4+0.85x0.4)x0.4+(0.7x0.3+0.72x0.3)x0.3+(0.8x0.3+0.78x0.3)x0.3=0.774.

[0076] Step S400: Based on the matching degree, determine the information retention of the target region in the first lightweight encoded image after lightweight encoding;

[0077] Specifically, when the matching degree of the current environment and the template is higher than the set threshold, according to the target region features in the established multi-source data feature template, DBSCAN algorithm is used in the point cloud data in the current lidar data by setting the neighborhood radius and minimum point number parameters to locate the target region; combined with the target position provided by the millimeter wave radar, the coordinate conversion and matching algorithm is used to associate the target position detected by the millimeter wave radar with the target region in the point cloud data, and the accuracy of the target region is confirmed; the located target region is verified by multi-source data cross verification; according to the target region contour information in the multi-source data feature template, the contour retention of the target region in the first lightweight encoded image is evaluated, and the specific method is: key points are extracted on the contour of the target region in the template and the contour of the first lightweight encoded image, the contour similarity is obtained by calculating the Euclidean distance between the key points, and when the contour similarity is higher than the set threshold, it is judged that the contour retention is normal; the color feature and shape feature of the target object are quantitatively evaluated, and the specific method is: for color feature, the color histogram is used to compare the color of the target object in the first lightweight encoded image with the color in the template; for shape feature, the shape matching algorithm based on Hu moment is used to calculate the similarity of the shape of the target object in the first lightweight encoded image and the shape of the template;

[0078] When the matching degree of the current environment and the template is lower than the set threshold, differences between the multi-source data features and the multi-source data feature template are analyzed to obtain abnormal data features; according to the abnormal data features, a Canny algorithm is used in the first lightweight coded image, combined with morphological processing, to find the object contour; combined with laser radar data and millimeter wave radar data, a data fusion algorithm based on Kalman filtering is used to determine the position and range of the target region; a topological structure matching algorithm based on contour shape is used to calculate the contour retention; through a data fusion algorithm based on Bayesian network, the current urban security monitoring multi-source data is fused, and according to the probability relationship between the current urban security monitoring multi-source data, the information retention of the target region is judged.

[0079] In a specific embodiment, the matching degree threshold of the current environment and the template is set to 0.8, higher than this threshold is high matching degree, and lower is low matching degree. In the morning period scene on the main road on a sunny day, the average matching degree of the current environment and the template is 0.85, which is higher than the threshold. According to the multi-source data feature template, DBSCAN algorithm is used in the current laser radar point cloud data to successfully locate multiple target regions. For example, a point cloud cluster is detected in a certain region, and it is calculated that the point cloud cluster meets the vehicle target region characteristics. Combined with the target position information provided by the millimeter wave radar, the coordinate conversion and matching algorithm is used to associate the target position 70 meters in front detected by the millimeter wave radar with the corresponding region in the laser radar point cloud data, and confirm that the target region is a vehicle in motion. Through cross verification of multi-source data, that is, comparing the visual features of the region in the image with the laser radar and millimeter wave radar data, the accuracy of the target region is confirmed, and in 100 positioning, the target region is accurately confirmed 95 times, with an accuracy of 95%. Key points such as inflection points and points with large curvature changes are extracted on the vehicle contour in the template and the vehicle contour in the first lightweight coded image. By calculating the Euclidean distance between these key points, the contour similarity is obtained. The vehicle target region in 100 first lightweight coded images in this scene is calculated, and the average contour similarity is 0.88, which is higher than the set threshold 0.75, and the contour retention is judged to be normal. For example, the average Euclidean distance between the vehicle contour key points in a first lightweight coded image and the template contour key points is 4 pixels (the image resolution is 1920x1080), and the contour similarity is calculated to be 0.9.

[0080] The color histogram is used to compare the color of the vehicle target object in the first lightweight encoded image with the color in the template. The Bhattacharyya distance of the color histogram is calculated, and the average Bhattacharyya distance of 100 images is 0.13, indicating that the color feature is well preserved. For example, the Bhattacharyya distance between the color histogram of a certain vehicle in the template and the color histogram in the first lightweight encoded image is 0.11, indicating that the color feature similarity is high. The shape matching algorithm based on Hu moments is used to calculate the shape similarity of the vehicle target object in the first lightweight encoded image and the template. After calculation, the average shape similarity of 100 images is 0.86, indicating that the shape feature is well preserved. For example, the Euclidean distance between the Hu moment feature vector of the vehicle shape in a certain image and the Hu moment feature vector of the vehicle shape in the template is small, and the shape similarity is calculated as 0.88.

[0081] In the rain, the matching degree of the current environment and the template in the crossroad scene at noon is 0.72 on average, which is lower than the threshold. By analyzing the differences between the multi-source data features and the template, it is found that the point cloud density in some areas is abnormally reduced due to rain interference, and the target object speed and distance information detected by the millimeter wave radar fluctuates greatly. In the first lightweight encoded image, the Canny algorithm is used in combination with morphological processing to find the object contour. The topological structure matching algorithm based on contour shape is used to calculate the contour preservation. This algorithm focuses on the connectivity, hole number, and general shape of the contour. The average topological structure similarity of 100 target regions in the first lightweight encoded image is 0.73, indicating that the contour preservation is acceptable. The multi-source data of the current urban security monitoring is fused by the data fusion algorithm based on Bayesian network. According to the probability relationship between the multi-source data of the current urban security monitoring, the information preservation of the target region is judged. In the evaluation of 100 target regions, 70 regions are judged to have good information preservation, accounting for about 70%.

[0082] Step S500: based on the information preservation, the first lightweight encoded image is adjusted, and after adjustment, the image is encoded according to the selected image encoding algorithm to generate the final lightweight encoded image.

[0083] Specifically, based on the information preservation, contour repair, color adjustment, shape adjustment, and texture adjustment are performed; among them, the contour repair uses the image repair algorithm based on PatchMatch, the color adjustment uses the histogram matching method, the shape adjustment uses the affine transformation method, and the texture adjustment uses the texture synthesis algorithm based on samples; the parameters are optimized according to the characteristics of the adjusted image, and the adjusted image is encoded according to the selected image encoding algorithm to generate the final lightweight encoded image.

[0084] For example, Fig. 2A system structure diagram of a deep learning-based image lightweight coding system is shown, and the present application provides a deep learning-based image lightweight coding system, comprising:

[0085] The comprehensive environment recognition module comprises a deep learning model training unit and a comprehensive environment recognition unit; wherein the deep learning model training unit collects historical urban security monitoring multi-source data, uses a convolutional neural network to train a deep learning model, and performs model training, evaluation and optimization; the comprehensive environment recognition unit recognizes the current comprehensive environment;

[0086] The lightweight coding image transfer module comprises a simplified image generation unit and a first lightweight coding image generation unit; wherein the simplified image generation unit constructs a generator and a discriminator based on the current comprehensive environment, adopts adversarial training, alternately trains the generator and the discriminator, performs coding optimization of the current security monitoring image, and generates a simplified image; the first lightweight coding image generation unit inputs the simplified image into an image coding algorithm for coding to obtain a first lightweight coding image;

[0087] The matching degree calculation module comprises a multi-source data feature extraction unit, a feature template establishment unit and a matching degree calculation unit; wherein the multi-source data feature extraction unit collects current urban security monitoring multi-source data and extracts multi-source data features; the feature template establishment unit establishes multi-source data feature templates under different comprehensive environments based on the current comprehensive environment, compares and analyzes the multi-source data features and the corresponding multi-source data feature templates, and obtains the matching degree of the environment and the template;

[0088] The information retention condition calculation module comprises an information retention condition calculation unit; wherein the information retention condition calculation unit determines the information retention condition of the target area in the first lightweight coding image after lightweight coding based on the matching degree;

[0089] The lightweight coding image generation module comprises a lightweight coding image adjustment unit and a lightweight coding image generation unit; wherein the lightweight coding image adjustment unit adjusts the first lightweight coding image based on the information retention condition, and after adjustment, the lightweight coding image generation unit codes according to the selected image coding algorithm to generate the final lightweight coding image.

[0090] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims with respect to the figures of the patent document.

Claims

1. A deep learning-based image lightweight encoding method, characterized in that, The application comprises the following steps: Collecting historical urban security monitoring multi-source data, training a deep learning model using a convolutional neural network, model training, evaluation and optimization, and identifying the current comprehensive environment; Based on the current comprehensive environment, a generator and a discriminator are constructed, and an adversarial training is adopted to alternately train the generator and the discriminator, and the current security monitoring image is encoded and optimized to generate a simplified image; The simplified image is input into the image encoding algorithm for encoding to obtain a first lightweight encoded image; An encoder-decoder structure is used as the basic framework of the generator, and the image features of the current security monitoring image are extracted through the encoder; The environment category of the current comprehensive environment is encoded into a one-hot vector, which is concatenated with the image features extracted by the encoder and input into the generator as a condition; a skip connection is added between the encoder and the decoder of the generator to connect the features at different levels in the encoder to the corresponding levels in the decoder; and the generator generates a simplified image to be detected; A discriminator based on a convolutional neural network is constructed, the input of the discriminator is the simplified image to be detected and the current security monitoring image, the discriminator is composed of multiple convolutional layers, and the features of the image are extracted through convolution operation; In the discriminator, multi-scale feature fusion is used to discriminate the authenticity and quality of the image; the output layer of the discriminator is a binary classification layer, which uses a sigmoid activation function to output a value between 0 and 1, representing the probability that the simplified image to be detected is the current security monitoring image; Prepare the training set, initialize the generator and the discriminator, and perform adversarial training to generate a simplified image; The training set uses historical urban security monitoring multi-source data to initialize the weight parameters of the generator and the discriminator, and the generator and the discriminator are compiled in the TensorFlow framework; the number of adversarial training rounds of the generator and the discriminator is determined; During the training process, the loss value changes of the generator and the discriminator are monitored in real time to determine whether they converge; the current security monitoring image and the corresponding current comprehensive environment are input into the generator, and the generator outputs a simplified image; Collecting current urban security monitoring multi-source data and extracting multi-source data features; Based on the current comprehensive environment, a multi-source data feature template under different comprehensive environments is established, and the multi-source data features are compared and analyzed with the corresponding multi-source data feature templates to obtain the matching degree of the environment and the template; Based on the matching degree, the information retention of the target region in the first lightweight encoded image after lightweight encoding is determined; Based on the information retention, the first lightweight encoded image is adjusted, and after adjustment, the image is encoded according to the selected image encoding algorithm to generate a final lightweight encoded image.

2. The image lightweight coding method based on deep learning according to claim 1, characterized in that, The application comprises the following steps: Collect images under different weather, different time and different scene from urban street cameras, collect laser radar data and millimeter wave radar data synchronously; the laser radar data provides three-dimensional point cloud information of the environment around the street, and the millimeter wave radar data includes distance, speed and angle information of target objects; integrate to form historical urban security monitoring multi-source data; Preprocess the historical urban security monitoring multi-source data, and divide it into training set, validation set and test set; the convolutional neural network structure is selected as ResNet, and the loss function, optimizer and training rounds are set; input the image data and corresponding sensor data in the training set into the deep learning model, the model calculates the prediction result through forward propagation, and calculates the loss value using the loss function according to the difference between the prediction result and the true label; calculate the gradient of the loss value to the model parameters through the back propagation algorithm, and update the model parameters using the optimizer, so that the prediction result of the model gradually approaches the true label; after each round of training, evaluate the performance of the model on the validation set, and record the loss value and accuracy on the validation set; model evaluation and optimization; Real-time collection of image data and sensor data around urban streets, preprocessed and input into a deep learning model, and the model outputs the current comprehensive environment.

3. The image lightweight coding method based on deep learning according to claim 1, characterized in that, The simplified image is input into an image encoding algorithm for encoding to obtain a first lightweight encoded image, comprising: The user specifies the image encoding algorithm, sets the encoding parameters, and performs data format conversion, including color space conversion and image size adaptation; encode the simplified image according to the encoding process of different encoding algorithms, and perform MD5 check after encoding to obtain the first lightweight encoded image.

4. The image lightweight coding method based on deep learning according to claim 1, characterized in that, The current urban security monitoring multi-source data is collected, and the multi-source data features are extracted, including: Collect current urban security monitoring multi-source data, including current security monitoring images, current laser radar data and current millimeter wave radar data; extract features of current security monitoring images, including color features, texture features and shape features, extract features of current laser radar data, including geometric features and motion features, the geometric features include point cloud density features, object geometric shape features and surface normal vector features, extract features of current millimeter wave radar data, including target object attribute features and target object motion mode features, the target object attribute features include distance features, speed features and angle features.

5. The image lightweight coding method based on deep learning according to claim 1, characterized in that, Based on the current comprehensive environment, a multi-source data feature template under different comprehensive environments is established, and the multi-source data features are compared and analyzed with the corresponding multi-source data feature templates to obtain the matching degree of the environment and the template, including: Based on the comprehensive environment-based environmental category, statistical analysis is performed to obtain multi-source data feature templates, including security monitoring image feature templates, laser radar data feature templates, and millimeter wave radar data feature templates; the current security monitoring image features, current laser radar data features, and current millimeter wave radar data features in the multi-source data features are compared with the security monitoring image feature templates, laser radar data feature templates, and millimeter wave radar data feature templates respectively, a comprehensive similarity score is calculated by setting the weights of different comparison results; and the matching degree of the current environment and the template is determined according to the comprehensive similarity score.

6. The image lightweight coding method based on deep learning according to claim 1, characterized in that, The matching degree is used to determine the information retention of the target region in the first lightweight encoded image after lightweight encoding, including: When the matching degree of the current environment and the template is higher than a set threshold, the DBSCAN algorithm is used on the point cloud data in the current laser radar data according to the target region features in the established multi-source data feature templates by setting the neighborhood radius and minimum point number parameters to locate the target region; the target position detected by the millimeter wave radar is associated with the target region in the point cloud data by using coordinate conversion and matching algorithms to confirm the accuracy of the target region in combination with the target position provided by the millimeter wave radar; the accuracy of the located target region is cross-verified by multi-source data; the contour retention of the target region in the first lightweight encoded image is evaluated according to the target region contour information in the multi-source data feature template, specifically by extracting key points on the contour of the target region in the template and the contour of the first lightweight encoded image, calculating the Euclidean distance between the key points to obtain a contour similarity, and determining that the contour retention is normal when the contour similarity is higher than a set threshold; the color features and shape features of the target object are quantitatively evaluated, specifically by comparing the color of the target object in the first lightweight encoded image with the color in the template using a color histogram for color features, and calculating the similarity of the shape of the target object in the first lightweight encoded image with the shape in the template using a shape matching algorithm based on Hu moments for shape features; When the matching degree of the current environment and the template is lower than a set threshold, the differences between the multi-source data features and the multi-source data feature templates are analyzed to obtain abnormal data features; the Canny algorithm is used in the first lightweight encoded image in combination with morphological processing to find the object contour according to the abnormal data features; the position and range of the target region are determined by using a data fusion algorithm based on Kalman filtering in combination with laser radar data and millimeter wave radar data; the contour retention is calculated by using a topological structure matching algorithm based on contour shape; the current urban security monitoring multi-source data is fused by using a data fusion algorithm based on Bayesian networks, and the information retention of the target region is determined according to the probability relationship between the current urban security monitoring multi-source data.

7. The image lightweight coding method based on deep learning according to claim 1, characterized in that, The first lightweight encoded image is adjusted based on the information retention, and the adjusted image is encoded according to the selected image encoding algorithm to generate a final lightweight encoded image, including: Based on the information retention, the contour repair, color adjustment, shape adjustment and texture adjustment are performed; wherein, the contour repair uses the image repair algorithm based on PatchMatch, the color adjustment uses the method based on histogram matching, the shape adjustment uses the affine transformation method, and the texture adjustment uses the sample-based texture synthesis algorithm; according to the characteristics of the adjusted image, the parameters are optimized, the adjusted image is encoded according to the selected image encoding algorithm, and the final lightweight encoded image is generated.

8. A deep learning-based image lightweight coding system using the deep learning-based image lightweight coding method of any one of claims 1-7. Comprehensive environment identification module: comprising: a deep learning model training unit and a comprehensive environment identification unit; wherein the deep learning model training unit collects historical city security monitoring multi-source data, trains a deep learning model using a convolutional neural network, performs model training, evaluation and optimization, and the comprehensive environment identification unit identifies the current comprehensive environment; Lightweight encoded image transfer module: comprising: a simplified image generation unit and a first lightweight encoded image generation unit; wherein the simplified image generation unit constructs a generator and a discriminator based on the current comprehensive environment, adopts adversarial training, alternately trains the generator and the discriminator, performs encoding optimization of the current security monitoring image, and generates a simplified image; the first lightweight encoded image generation unit inputs the simplified image into the image encoding algorithm for encoding to obtain a first lightweight encoded image; Matching degree calculation module: comprising: a multi-source data feature extraction unit, a feature template establishment unit and a matching degree calculation unit; wherein the multi-source data feature extraction unit collects current city security monitoring multi-source data and extracts multi-source data features; the feature template establishment unit establishes multi-source data feature templates under different comprehensive environments based on the current comprehensive environment, compares and analyzes the multi-source data features and the corresponding multi-source data feature templates, and obtains the matching degree of the environment and the template; Information retention condition calculation module: comprising: an information retention condition calculation unit; wherein the information retention condition calculation unit determines the information retention condition of the target region in the first lightweight encoded image after lightweight encoding based on the matching degree; Lightweight encoded image generation module: comprising: a lightweight encoded image adjustment unit and a lightweight encoded image generation unit; wherein the lightweight encoded image adjustment unit adjusts the first lightweight encoded image based on the information retention condition, and after adjustment, the lightweight encoded image generation unit encodes according to the selected image encoding algorithm to generate the final lightweight encoded image. ​

Images