Data processing method, device, equipment and medium for machine vision

Abstract

The embodiments of the present application provide a data processing method, device, equipment and medium for machine vision, which belongs to the field of visual data processing technology. The method includes: inputting the acquired initial visual information into a trained end-side compression model, performing different types of feature extraction on the initial visual information through a multi-level encoder, and obtaining the corresponding types of coding features at each level; performing vector quantization processing on each coding feature based on a visual code book to obtain the corresponding type of compressed information, so that after receiving the compressed information of each type, the edge-cloud side system reconstructs the coding features represented by the different types of compressed information through a multi-level decoder, obtains the corresponding types of decoding features at each level, and then obtains the target visual information that matches the initial visual information. The present application can improve the transmission efficiency of visual data.

Description

Technical Field

[0001] The present application relates to the technical field of visual data processing, and in particular to a data processing method, device, equipment and medium for machine vision. Background Art

[0002] Visual data refers to data in visual forms such as images and videos. With the rapid development of digital information, there is an increasing need to efficiently transfer massive amounts of visual data from one place to another, especially to transfer visual data from end-side devices to edge / cloud-side processing systems.

[0003] In related technologies, the original visual data to be transmitted (such as videos, images, etc.) is compressed to obtain continuous visual features, and the original visual data is transmitted from the end-side device to the edge / cloud-side processing system based on the continuous visual features. However, due to the limitation of communication bandwidth, it is difficult for the end-side and edge-cloud side to gather a large amount of compressed data in real time, which leads to low transmission efficiency of visual data. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a data processing method, device, equipment and medium for machine vision, aiming to improve the transmission efficiency of visual data.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a data processing method for machine vision, which is applied to a terminal device. The terminal device and the edge-cloud system are both provided with the same visual codebook. The data processing method for machine vision includes:

[0006] Acquiring initial visual information to be transmitted;

[0007] The initial visual information is input into a pre-trained end-side compression model, and different types of features are extracted from the initial visual information through a multi-level encoder to obtain the corresponding types of encoding features at each level.

[0008] Based on the visual codebook, vector quantization is performed on the coding features of each type to obtain the corresponding type of compressed information;

[0009] The corresponding types of compressed information are sent to the edge-cloud system, so that the edge-cloud system inputs the compressed information of each type into the pre-trained edge-cloud decoding model, and reconstructs the encoding features representing different types of compressed information through a multi-level decoder to obtain the corresponding types of decoding features at each level; and fuses the decoding features of different types to obtain the target visual information that matches the initial visual information.

[0010] In some embodiments, the multi-level encoder includes a shallow level coding block and a deep level coding block; the coding features include texture coding features and semantic coding features;

[0011] The initial visual information is subjected to different types of feature extraction through a multi-level encoder to obtain the corresponding types of encoding features at each level, including:

[0012] Perform texture encoding on the initial visual information based on the shallow coding block to obtain the texture coding features under the texture type;

[0013] The texture coding features are semantically encoded based on deep coding blocks to obtain semantic coding features under semantic types.

[0014] In some embodiments, the visual codebook includes a plurality of code vectors identified by code vector indices;

[0015] Based on the visual codebook, vector quantization is performed on the coding features of each type to obtain the corresponding type of compressed information, including:

[0016] Performing dimensionality reduction processing on the texture coding feature to obtain a plurality of first basic feature units after the texture coding feature is reduced in dimension;

[0017] Determine a first code vector matching the first basic feature unit from the multiple code vectors, and use a code vector index corresponding to the first code vector as compression information under the texture type;

[0018] Alternatively, multiple second basic feature units of the semantic coding feature are determined, a second code vector matching the second basic feature unit is determined from multiple code vectors, and a code vector index corresponding to the second code vector is used as compression information under the semantic type.

[0019] In some embodiments, the visual codebook is obtained by the following steps, comprising:

[0020] Acquire initial visual information of the sample to be transmitted;

[0021] Input the sample's initial visual information into a pre-built end-side compression model, and use a multi-level encoder to extract different types of features from the sample's initial visual information to obtain sample encoding features of the corresponding type at each level.

[0022] Based on the preset initial codebook, vector quantization is performed on the sample coding features of each type to obtain the sample compression information of the corresponding type;

[0023] Based on the sample compression information, a preset first gradient stop operator and a preset first weight factor, a first quantization loss value is calculated;

[0024] The initial codebook is updated according to the first quantization loss value to obtain a visual codebook.

[0025] To achieve the above-mentioned purpose, the second aspect of the embodiment of the present application proposes a data processing method for machine vision, which is applied to the edge-cloud system. The same visual codebook is set in both the terminal device and the edge-cloud system. The data processing method for machine vision includes:

[0026] Receive compressed information of various types sent from the end-side device, where the compressed information is obtained by performing vector quantization processing on coding features of various types by the end-side device based on a visual codebook; the coding features of various types are obtained by inputting the acquired initial visual information into a pre-trained end-side compression model and extracting different types of features from the initial visual information through a multi-level encoder;

[0027] Input each type of compressed information into the pre-trained edge-cloud decoding model, reconstruct the encoding features representing different types of compressed information through a multi-level decoder, and obtain the corresponding type of decoding features at each level;

[0028] The decoded features of different types are fused to obtain the target visual information that matches the initial visual information.

[0029] In some embodiments, the visual codebook is obtained by the following steps, comprising:

[0030] Acquire initial visual information of the sample to be transmitted;

[0031] Input the sample's initial visual information into a pre-built end-side compression model, and use a multi-level encoder to extract different types of features from the sample's initial visual information to obtain sample encoding features of the corresponding type at each level.

[0032] Based on the preset initial codebook, vector quantization is performed on the sample coding features of each type to obtain the sample compression information of the corresponding type;

[0033] Based on the sample compression information, a preset first gradient stop operator and a preset first weight factor, a first quantization loss value is calculated;

[0034] The initial codebook is updated according to the first quantization loss value to obtain a visual codebook.

[0035] In some embodiments, the edge cloud decoding model is trained by the following steps, including:

[0036] Receive sample compression information of various types sent by the end-side device, where the sample compression information is obtained by the end-side device performing vector quantization processing on sample coding features of various types based on a visual codebook; the sample coding features of various types are obtained by inputting the acquired sample initial visual information into a pre-trained end-side compression model and performing different types of feature extraction on the sample initial visual information through a multi-level encoder;

[0037] The sample compression information of each type is input into the pre-built edge cloud decoding model, and the sample encoding features represented by the different types of sample compression information are reconstructed through a multi-level decoder to obtain the sample decoding features of the corresponding types at each level;

[0038] Fusion of sample decoding features of different types to obtain sample target visual information that matches the sample initial visual information;

[0039] Based on the sample initial visual information and the sample target visual information, the pixel loss value and the perceptual loss value are calculated. The edge-cloud side parameters of the edge-cloud side decoding model are adjusted according to the pixel loss value and the perceptual loss value to obtain a trained edge-cloud side decoding model.

[0040] In some embodiments, the terminal-side compression model and the edge-cloud-side decoding model are jointly trained by the following steps, including:

[0041] Based on a preset second gradient stop operator and a second weight factor, a second quantization loss value of a sample encoding feature and a sample decoding feature of the same type is calculated;

[0042] The end-side parameters and the edge-cloud side parameters are jointly adjusted according to the second quantization loss value to obtain a trained end-side compression model and an edge-cloud side decoding model.

[0043] In some embodiments, the terminal-side compression model and the edge-cloud-side decoding model are jointly trained by the following steps, which also include:

[0044] Based on the discriminator introduced in advance, the sample initial visual information is discriminated to obtain a first discriminant value, and the sample target visual information is discriminated to obtain a second discriminant value;

[0045] The logarithmic processing result of the first discriminant value and the logarithmic processing result of the second discriminant value are superimposed to obtain the adversarial loss value, and the end-side parameters and the edge-cloud side parameters are jointly adjusted according to the adversarial loss value to obtain the trained end-side compression model and edge-cloud side decoding model.

[0046] In some embodiments, the terminal-side compression model and the edge-cloud-side decoding model are jointly trained by the following steps, which also include:

[0047] Based on the pre-introduced teacher model, feature extraction is performed on the initial visual information of the sample to obtain the sample prediction decoding features;

[0048] Calculate the mean square error between the sample prediction decoding features and the sample decoding features to obtain the distillation loss value;

[0049] The end-side parameters and edge-cloud-side parameters are jointly adjusted according to the distillation loss value to obtain the trained end-side compression model and edge-cloud-side decoding model.

[0050] In some embodiments, the terminal-side compression model and the edge-cloud-side decoding model are jointly trained by the following steps, which also include:

[0051] According to the preset training weights, the first quantization loss value, the pixel loss value, the perceptual loss value, the second quantization loss value, the adversarial loss value and the distillation loss value are superimposed to obtain a total loss value;

[0052] The end-side parameters and edge-cloud side parameters are jointly adjusted according to the total loss value to obtain the trained end-side compression model and edge-cloud side decoding model.

[0053] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application proposes a data processing device for machine vision, in which a visual codebook is provided, and the device includes:

[0054] An acquisition module, used for acquiring initial visual information to be transmitted;

[0055] The feature encoding module is used to input the initial visual information into the pre-trained end-side compression model, extract different types of features from the initial visual information through a multi-level encoder, and obtain the corresponding type of encoding features at each level;

[0056] A vector quantization module is used to perform vector quantization processing on the coding features of each type based on the visual codebook to obtain the corresponding type of compression information;

[0057] The target transmission module is used to send the corresponding type of compressed information to the edge-cloud side system, so that the edge-cloud side system inputs the compressed information of each type into the pre-trained edge-cloud side decoding model, and reconstructs the encoding features representing different types of compressed information through a multi-level decoder to obtain the corresponding type of decoding features at each level; and fuses the decoding features of different types to obtain the target visual information that matches the initial visual information.

[0058] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and when the processor executes the computer program, the data processing method for machine vision of the first aspect or the data processing method for machine vision of the second aspect is implemented.

[0059] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the data processing method for machine vision of the first aspect or the data processing method for machine vision of the second aspect.

[0060] The data processing method, device, equipment and medium for machine vision proposed in the present application obtain the initial visual information to be transmitted; input the initial visual information into a pre-trained edge-side compression model, and extract different types of features from the initial visual information through a multi-level encoder to obtain the corresponding type of coding features at each level; then, based on the visual codebook, respectively perform vector quantization processing on the coding features of each type to obtain the corresponding type of compressed information; send the corresponding type of compressed information to the edge-cloud system, so that the edge-cloud system inputs the compressed information of each type into a pre-trained edge-cloud decoding model, and reconstructs the coding features represented by the different types of compressed information through a multi-level decoder to obtain the corresponding type of decoding features at each level; and fuses the decoding features of different types to obtain the target visual information that matches the initial visual information; and, since the present application transmits the compressed information corresponding to the initial visual information determined based on the visual codebook, the amount of information that needs to be transmitted is reduced, so that the edge-cloud system can reconstruct the final target visual information based on the small amount of information transmitted by the edge-side device, thereby improving the transmission efficiency of visual data.‌ BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a schematic diagram of an optional application scenario of the data processing device for machine vision provided in an embodiment of the present application;

[0062] Figure 2 is an optional flow chart of a data processing method for machine vision provided in an embodiment of the present application;

[0063] Figure 3 It is an optional transmission schematic diagram of the data processing method for machine vision provided in the embodiment of the present application;

[0064] Figure 4 yes Figure 2 A flowchart of an implementation of step 102 in FIG.

[0065] Figure 5 yes Figure 2 A flowchart of an implementation of step 103 in FIG.

[0066] Figure 6 is another optional flow chart of the data processing method for machine vision provided in an embodiment of the present application;

[0067] Figure 7 It is an optional visual codebook update flow chart of the data processing method for machine vision provided in the embodiment of the present application;

[0068] Figure 8 It is an optional model training schematic diagram of the data processing method for machine vision provided in the embodiment of the present application;

[0069] Fig. 9 It is an optional edge-cloud decoding model training flow chart of the machine vision-oriented data processing method provided in an embodiment of the present application;

[0070] Fig.10 It is an optional joint training flow chart of the data processing method for machine vision provided in the embodiment of the present application;

[0071] Fig.11 is another optional joint training flow chart of the data processing method for machine vision provided in an embodiment of the present application;

[0072] Fig.12 It is another optional joint training flow chart of the data processing method for machine vision provided in the embodiment of the present application;

[0073] Fig.13 It is another optional joint training flow chart of the data processing method for machine vision provided in the embodiment of the present application;

[0074] Fig.14 This is an optional flow chart of a data processing device for machine vision provided in an embodiment of the present application;

[0075] Fig.15 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0077] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0079] Visual data refers to data in visual forms such as images and videos. In today's digital age, visual sensors are widely used in many scenarios such as public transportation, public safety, industrial production and commercial services, which has generated large-scale and complex visual data. In particular, especially in the context of the rapid development of smart cities, visual data such as ultra-high-resolution images and videos generated by smart terminals have brought great challenges to the real-time transmission and perception processing of large-scale data.

[0080] In related technologies, the original visual data to be transmitted (such as videos, images, etc.) is compressed to obtain continuous visual features, and the original visual data is transmitted from the end-side device to the edge / cloud-side processing system based on the continuous visual features. However, due to the limitation of communication bandwidth, it is difficult for the end-side and edge-cloud side to gather a large amount of compressed data in real time, which leads to low transmission efficiency of visual data.

[0081] Based on this, the embodiments of the present application provide a data processing method, device, equipment and medium for machine vision, aiming to improve the transmission efficiency of visual data.

[0082] It should be noted that in the embodiments of the present application, when it comes to information related to user characteristics such as user basic information or user identity, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or consent will be obtained first. After the user's separate permission or consent is clearly obtained, the necessary data for the normal operation of the embodiments of the present application will be obtained. For example, when the terminal side device of the embodiments of the present application obtains the initial visual information to be transmitted, it will first obtain the consent of the relevant user, otherwise it will not be able to obtain the initial visual information and perform subsequent data processing operations. In addition, the other relevant data obtained by the embodiments of the present application are all authorized data after obtaining consent, which will not be repeated here.

[0083] The data processing method, device, equipment and medium for machine vision provided in the embodiments of the present application are specifically described through the following embodiments. First, the application scenario of the data processing device for machine vision in the embodiments of the present application is described, such as Figure 1 As shown, Figure 1 : is an optional application scenario diagram of a data processing device for machine vision provided in an embodiment of the present application, wherein the data processing device for machine vision in the embodiment of the present application includes an end-side device and an edge-cloud device, and the end-side compression model of the end-side device includes the following modules:

[0084] Data acquisition module: used to acquire initial visual information to be transmitted from multiple information terminals (including but not limited to mobile phone cameras, drone cameras, etc.);

[0085] Feature encoding module: used to input the acquired initial visual information into the pre-trained end-side compression model, extract different types of features from the initial visual information through a multi-level encoder, and obtain the corresponding type of encoding features (continuous features) at each level;

[0086] Vector quantization module: used to perform vector quantization processing on the coding features of each type based on the visual codebook to obtain the corresponding type of compressed information (discrete features);

[0087] Target transmission module: used to send the corresponding type of compressed information to the edge cloud side device.

[0088] The edge-cloud decoding and compression model of the edge-cloud device includes the following modules:

[0089] Target transmission module: used to receive compressed information (discrete features) of various types sent from the end-side device;

[0090] Vector quantization module: used to input compressed information of various types into the pre-trained edge-cloud decoding model, reconstruct the encoding features represented by different types of compressed information through a multi-level decoder, and obtain the corresponding type of decoding features (continuous features) at each level;

[0091] Feature encoding module: used to fuse decoding features of different types;

[0092] Downstream task module: used to further process the decoded features to obtain target visual information that matches the initial visual information; illustratively, Figure 1When the downstream task is image recognition, the image recognition information is output; when the downstream task is target detection, the target detection information is output; when the downstream task is sample segmentation, the sample segmentation information is output; wherein the image recognition information, the target detection information and the sample segmentation information are all target visual information that matches the initial visual information. The downstream task can be adaptively adjusted according to the actual situation, and the embodiments of the present application do not limit this.

[0093] After a preliminary understanding of the example application scenarios of the data compression and transmission device proposed in this application, the data processing method for machine vision proposed in the embodiment of this application will be explained next, and the beneficial effects brought about by the data processing method for machine vision will gradually become apparent.

[0094] In the embodiment of the present application, the description will be made from the perspective of the device deployed on the end-side device, wherein the same visual codebook is set in both the end-side device and the edge-cloud system. Figure 2 As shown, Figure 2 is an optional flowchart of a data processing method for machine vision provided in an embodiment of the present application. Figure 2 The method may include but is not limited to the following steps 101 to 104. When the device deployed on the end-side device executes the data processing method for machine vision, the specific process is as follows. It should be noted that this embodiment Figure 2 The order of step 101 to step 104 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0095] Step 101: Acquire initial visual information to be transmitted.

[0096] The following is a detailed description of step 101.

[0097] Among them, the end-side device refers to the device located at the edge of the network in the end-edge-cloud architecture. The end-side device has certain computing power, storage capacity and network connection capabilities to perform preliminary processing and intelligent decision-making on local data. End-side devices include but are not limited to smartphones, computers, smart home devices, wearable devices, self-driving cars, industrial equipment, smart security cameras, smart headphones, etc.

[0098] Furthermore, the end-edge-cloud architecture is used to distribute data processing, storage, and management functions to the edge devices and cloud computing platforms of the network to improve data processing efficiency, reduce latency, and enhance user experience. The end-edge-cloud architecture mainly consists of three components: ① End devices, such as sensors, cameras, etc., responsible for data collection; ② Edge computing, which processes data closer to the user; ③ Cloud computing, which performs more complex data analysis and storage. Usually, edge computing and cloud computing can be combined to form an edge-cloud system in order to fully utilize the advantages of both and achieve more efficient, more secure, and lower latency computing services.

[0099] Among them, the end-edge-cloud architecture in the embodiment of the present application can be constructed based on networks such as Convolutional Neural Network (CNN), Vision Transformer, and Visual Representation Network Based on Heat Conduction (vHeat). Of course, the end-edge-cloud architecture can also be constructed based on other networks, and the embodiment of the present application does not limit this.

[0100] Furthermore, edge-cloud systems refer to systems that extend the capabilities of cloud computing to the edge of the network in an edge-edge-cloud architecture to enable local processing and analysis of data, improve application performance, reduce bandwidth requirements, and provide faster real-time insights. Edge-cloud systems include but are not limited to industrial Internet of Things, intelligent transportation systems, content distribution systems, mobile edge computing systems, smart home systems, smart city systems, etc.

[0101] Among them, the initial visual information can be obtained by real-time capture through a visual capture device, such as a static image taken by a digital camera or a mobile phone, a video captured by a surveillance camera, a driving recorder, a smart phone and other devices, an image of the earth's surface taken by a satellite, a medical image taken by a medical device, an infrared image captured by an infrared camera, etc.

[0102] Alternatively, the initial visual information can be obtained by the end-side device from a third-party information source. For example, the end-side device downloads and stores the initial visual information from an open database by connecting to the World Wide Web, and then transmits the initial visual information to the edge-cloud system based on the transmission method proposed in the embodiment of the present application. The target visual information reconstructed in the edge-cloud decoding model matches the initial visual information, that is, the target visual information is consistent with the initial visual information in terms of visual data type, information size, information content, etc.

[0103] Step 102: input the initial visual information into a pre-trained end-side compression model, and perform different types of feature extraction on the initial visual information through a multi-level encoder to obtain corresponding types of encoding features at each level.

[0104] The following is a detailed description of step 102.

[0105] Among them, Figure 3 As shown, Figure 3 is an optional transmission diagram of the data processing method for machine vision provided in the embodiment of the present application. For the sake of convenience, the embodiment of the present application only uses Figure 3 The example of the case where the initial visual information is an image does not limit the form of the initial visual information in the embodiments of the present application. Specifically, the initial visual information obtained in step 101 is input into a pre-trained end-side compression model, and a multi-level encoder is deployed in the end-side compression model, wherein the multi-level encoder is composed of multiple encoding blocks, and different encoding blocks can extract different types of features in the initial visual information, so as to perform data processing at different levels on the corresponding extracted features, and then obtain the corresponding type of encoding features at each level.

[0106] Furthermore, before the initial visual information is input into the client-side compression model, a series of preprocessing operations may be performed on the initial visual information, such as Figure 3 As shown, the initial visual information (initial image) to be transmitted is subjected to image segmentation processing to obtain multiple image blocks, so that the end-side compression model can perform subsequent processing on each image block separately. Of course, the preprocessing operation can also include size adjustment, data enhancement, noise removal, deblurring, etc. The specific preprocessing operation can be selected according to the actual situation, and the embodiment of the present application does not limit this.

[0107] It can be understood that the embodiments of the present application better retain different types of key features in the initial visual information by capturing different types of features from the initial visual information, thereby improving the completeness and richness of the features extracted from the initial visual information to be transmitted. In this way, while the terminal-side device realizes efficient transmission of visual data, there is no need to specifically combine downstream tasks. The transmitted complete, rich, and key features make the device of the embodiments of the present application more versatile and can better adapt to various application scenarios and needs.

[0108] In some embodiments, Figure 4 yes Figure 2 A flowchart for implementing step 102 in the embodiment of the present invention is provided, wherein different types of features are extracted from the initial visual information by a multi-level encoder to obtain coding features of corresponding types at each level, including the following steps 201 to 202:

[0109] Step 201 , texture encoding is performed on the initial visual information based on the shallow coding block to obtain texture coding features under the texture type.

[0110] Step 202: semantically encode the texture coding features based on the deep coding blocks to obtain semantic coding features under the semantic type.

[0111] Steps 201 to 202 are described in detail below.

[0112] In some embodiments, Figure 3 As shown, the multi-level encoder includes shallow-level coding blocks and deep-level coding blocks. Specifically, both shallow-level and deep-level coding blocks can contain several basic network layers such as convolutional layers or transformation layers (Transformer). The shallow-level coding block is used to extract low-level visual features from the initial visual information, and the deep-level coding block is used to further extract high-level visual features from the low-level visual features. Among them, the low-level visual features include but are not limited to visual texture features, visual color features, visual edge features, etc.; the deep-level visual features include but are not limited to visual spatial features, visual semantic features, etc.

[0113] Furthermore, the multi-level encoder included in the end-side compression model may also be a mid-level encoder, and the feature extraction capability of the mid-level encoder is between that of the shallow-level encoding block and the high-level encoder. The features extracted by the mid-level encoder may be visual shape features, etc., which is also not limited in the present embodiment.

[0114] For ease of understanding, Figure 3 As shown, the embodiment of the present application is described in detail by taking the example that the end-side compression model includes shallow-level coding blocks and deep-level coding blocks, and the shallow-level coding blocks are used to extract visual texture features and the deep-level coding blocks are used to extract visual semantic features.

[0115] Furthermore, if Figure 3 As shown, the shallow coding blocks in the edge-side compression model encode the texture features in the initial visual information to obtain texture coding features under the texture type, so that the texture coding features can be subsequently processed and transmitted to the edge-cloud side system; then, the texture coding features are further semantically encoded based on the deep coding blocks to obtain semantic coding features under the semantic type, so that the semantic coding features can be subsequently processed and transmitted to the edge-cloud side system.

[0116] In other embodiments, the initial visual information can also be input into the shallow coding block and the deep coding block respectively, and the feature extraction capability of the deep coding block is better than that of the shallow coding block. Therefore, the deep coding block does not need to use the output result of the shallow coding block as input, but directly performs feature extraction processing from the initial visual information to obtain the corresponding deep coding features.

[0117] Alternatively, the deep coding block takes the shallow coding features and the initial visual information as input, and assigns different processing weights to the shallow coding features and the initial visual information in the process of feature extraction based on the deep coding block, and then fuses the processing results of the encoders at each level to obtain the deep coding features. In addition, the processing weights assigned to the shallow coding features and the initial visual information can be set according to actual conditions.

[0118] Step 103: Based on the visual codebook, vector quantization processing is performed on the coding features of each type to obtain compression information of the corresponding type.

[0119] The following is a detailed description of step 103.

[0120] Among them, the codebook refers to a set of representative samples or code vectors. Specifically, vector quantization is achieved by mapping the input K-dimensional vector to a quantization vector in the codebook. The visual codebook refers to the same codebook deployed in both the end-side device and the edge-cloud system. In this way, continuous features are approximated to several code vectors, so that a set of codewords with a smaller storage share can be used to represent vector features with a larger storage share, reducing the amount of information that needs to be transmitted, thereby enabling the edge-cloud system to reconstruct decoding features that match the encoding features to be transmitted based on the small amount of information transmitted by the end-side device, thereby achieving efficient transmission of initial visual information.

[0121] In some embodiments, Figure 5 yes Figure 2 A flowchart for implementing step 103 in FIG. 1 is provided, wherein based on the visual codebook, vector quantization processing is performed on the coding features of each type to obtain compression information of the corresponding type, including the following steps 301 to 303:

[0122] Step 301, performing dimensionality reduction processing on the texture coding feature to obtain a plurality of first basic feature units after the texture coding feature is reduced in dimension.

[0123] Step 302: determine a first code vector matching the first basic feature unit from a plurality of code vectors, and use a code vector index corresponding to the first code vector as compression information under the texture type.

[0124] Step 303, alternatively, multiple second basic feature units of the semantic coding feature are determined, a second code vector matching the second basic feature unit is determined from multiple code vectors, and a code vector index corresponding to the second code vector is used as compression information under the semantic type.

[0125] Steps 301 to 303 are described in detail below.

[0126] In some embodiments, the visual codebook in the embodiments of the present application includes a visual texture codebook and a visual semantic codebook, each of which includes a plurality of code vectors identified by code vector indices, and the number of code vectors and the code vector dimension in the two codebooks may take different values. Figure 3 In the example, when the features extracted by the encoder are vectorized using the visual texture codebook, since the information representations between the features in the shallow dimension are similar, the extracted texture coding features can be firstly reduced in dimension. For example, a linear projection layer is introduced to map the texture coding features output by the encoder into a low-dimensional discrete codebook space (for example, from 1024 dimensions to 128 dimensions), and then vectorized in the low-dimensional discrete codebook space.

[0127] Further, multiple first basic feature units (tokens) are determined from the texture coding features after dimensionality reduction, and each first token is compared with each code vector in the visual texture codebook to find the most similar code vector as the quantization result (first code vector); then, the code vector index corresponding to the first code vector is determined as the compression information under the texture type, and the first code vector is used as the compression information under the texture type. Figure 3 In the example, the compression information under the texture type is the texture compression information.

[0128] Further, multiple second basic feature units are determined from the semantic coding features after dimensionality reduction, each second token is compared with each code vector in the visual semantic codebook, and the most similar code vector is found as the quantization result (second code vector); then, the code vector index corresponding to the second code vector is determined as the compression information under the semantic type, and the second basic feature units are compared with each code vector in the visual semantic codebook. Figure 3 In the example, the compression information under the semantic type is semantic compression information.

[0129] Furthermore, when the end-side compression model is trained, the visual texture codebook and the visual semantic codebook respectively contain a number of randomly initialized code vectors, and the code vectors are continuously updated during the training process until a trained end-side compression model is obtained and a determined visual codebook is obtained.

[0130] Step 104, sends the corresponding type of compressed information to the edge-cloud side system, so that the edge-cloud side system inputs the compressed information of each type into the pre-trained edge-cloud side decoding model, and reconstructs the encoding features representing the different types of compressed information through a multi-level decoder to obtain the corresponding type of decoding features at each level; and fuses the decoding features of different types to obtain the target visual information that matches the initial visual information.

[0131] Step 104 is described in detail below.

[0132] In some embodiments, after the end-side model obtains compressed information of various types, all compressed information is transmitted to the edge-cloud side system so that the edge-cloud side system can perform feature reconstruction processing on the encoded features represented by the compressed information, and then restore the target visual information that matches the initial visual information.

[0133] It can be understood that compressed information greatly reduces the amount of data compared to the initial visual information, thereby helping to reduce the demand for network bandwidth. Especially in an environment with limited bandwidth, the transmission device proposed in the embodiment of the present application can transmit visual data at an extremely low bit rate, thereby reducing transmission delay and improving transmission speed.

[0134] In the embodiment of the present application, the description will also be made from the dimension of the device deployed in the edge-cloud system, wherein the same visual codebook is set in both the end-side device and the edge-cloud system. Figure 6 As shown, Figure 6 is another optional flowchart of the data processing method for machine vision provided in an embodiment of the present application. Figure 6 The method may include but is not limited to the following steps 401 to 403. When the device of the edge-cloud system executes the data processing method for machine vision, the specific process is as follows. It should be noted that this embodiment Figure 6 The order of step 401 to step 403 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0135] Step 401, receiving compressed information of various types sent from the end-side device, wherein the compressed information is obtained by the end-side device by performing vector quantization processing on the coding features of each type based on the visual codebook; the coding features of each type are obtained by inputting the acquired initial visual information into a pre-trained end-side compression model, and performing different types of feature extraction on the initial visual information through a multi-level encoder.

[0136] In step 402, compressed information of each type is input into a pre-trained edge-cloud decoding model, and encoding features representing different types of compressed information are reconstructed through a multi-level decoder to obtain decoding features of the corresponding type at each level.

[0137] Step 403: Fusing the decoded features of different types to obtain target visual information that matches the initial visual information.

[0138] Steps 401 to 403 are described in detail below.

[0139] Among them, the specific implementation method of step 401 is similar to step 101 to step 103, which will not be repeated here. For the edge-cloud decoding device, after receiving compressed information of different types transmitted by the end-side device, it determines the code vector corresponding to each compressed information according to the visual code book set in the edge-cloud system that is the same as the end-side device. Since the code vector corresponds to the encoding features output by each encoder in the end-side compression model, and the edge-cloud decoding model also deploys multiple decoders that match the encoders in the end-side compression model, different decoders can reconstruct features of code vectors of different types, and then obtain decoding features of corresponding types at each level.

[0140] Further, in Figure 3 In the example, the edge-cloud decoding model includes shallow decoding blocks and deep decoding blocks, wherein the shallow decoding blocks correspond to shallow encoding blocks, and the deep decoding blocks correspond to deep encoding blocks. The shallow decoding block receives the texture compression information sent by the compression model on the end side, and determines the corresponding texture code vector based on the visual texture code book; then, the shallow decoding block performs feature reconstruction based on the texture code vector to obtain texture decoding features. Similarly, the deep decoding block receives the semantic compression information sent by the compression model on the end side, and determines the corresponding semantic code vector based on the visual semantic code book; then, the deep decoding block performs feature reconstruction based on the semantic code vector to obtain semantic decoding features.

[0141] Furthermore, a feature map is obtained by simply superimposing the decoding features of different types, or assigning different fusion weights to the decoding features of different types, and performing fusion processing according to the decoding features of each type and the corresponding fusion weights to obtain a feature map. The fused feature map is then input into the image reconstruction module corresponding to the downstream task to obtain the target visual information that matches the initial visual information.

[0142] Furthermore, the image reconstruction module can output the target visual information through one of the following two schemes:

[0143] Image reconstruction module based on convolution-upsampling: This scheme processes the input feature map through alternating stacked convolution layers-upsampling layers. Through repeated convolution and upsampling processes, the resolution of the feature map is constantly approaching the size of the original image. As upsampling proceeds, the originally sparse feature map becomes denser, and the image details are gradually restored, and finally an image with the same size as the initial visual information is reconstructed; Image block-by-image reconstruction method based on linear layers: This scheme uses linear layers to transform each token of the input feature map, predicts the image block corresponding to each token, and then splices these image blocks to obtain the target visual information. Furthermore, the image reconstruction module can be set according to the specific downstream tasks in actual applications. For example, when the downstream tasks are the following situations, the image reconstruction module can perform corresponding feature processing on the input feature map:

[0144] ① Image super-resolution task: In the image super-resolution task, the image reconstruction module can be designed as a super-resolution network to predict the high-resolution image using the decoding features reconstructed by the edge-cloud decoding model;

[0145] ② Image denoising task: In the image denoising task, the image reconstruction module can be designed as a denoising network to reconstruct a clean and noise-free image using the decoding features reconstructed by the edge-cloud decoding model;

[0146] ③ Video frame interpolation task: In the video frame interpolation task, the image reconstruction module can be designed as a frame interpolation network to generate intermediate frames using the decoding features reconstructed by the edge-cloud decoding model, thereby improving the frame rate of the final output video;

[0147] ④ Image segmentation task: In the image segmentation task, the image reconstruction module can be a segmentation network, which uses the decoding features reconstructed by the edge-cloud decoding model to determine the category label of each pixel in the transmitted image.

[0148] It should be noted that the feature processing capability of the image reconstruction module varies with the actual situation. The above four situations are only examples and the embodiments of the present application do not limit this.

[0149] After understanding how the transmission device of the present application processes the input initial visual information based on the transmission method proposed in the present application during the application reasoning process, the end-side compression model and edge-cloud side decoding model of the embodiment of the present application are then explained in detail. It can be understood that the end-side compression model and edge-cloud side decoding model will only be put into actual application reasoning after the training is completed. The application reasoning part of the device is first explained in the embodiment of the present application in order to let the reader first understand the value of the device in this case when it is put into actual application scenarios after the training is completed. It does not mean that the application of the device and the training sequence of the embodiment of the present application are restricted.

[0150] In some embodiments, Figure 7 It is an optional visual codebook update flow chart of the data processing method for machine vision provided in an embodiment of the present application, including the following steps 501 to 505:

[0151] Step 501, obtaining sample initial visual information to be transmitted.

[0152] Step 502: input the sample initial visual information into a pre-built end-side compression model, perform different types of feature extraction on the sample initial visual information through a multi-level encoder, and obtain sample coding features of corresponding types at each level.

[0153] Step 503: Based on a preset initial codebook, vector quantization processing is performed on sample coding features of each type to obtain sample compression information of the corresponding type.

[0154] Step 504: Calculate a first quantization loss value based on the sample compression information, a preset first gradient stop operator, and a preset first weight factor.

[0155] Step 505: Update the initial codebook according to the first quantization loss value to obtain a visual codebook.

[0156] Steps 501 to 505 are described in detail below.

[0157] In some embodiments, in order to improve data compression efficiency and optimize transmission quality, the end-side compression model needs to update the codebook used during the training process so that the trained end-side compression model can better perform vector quantization on the input initial visual information. The initial codebook refers to a basic codebook set set before starting the vector quantization process, and the initial codebook includes multiple code vectors (or code words) identified by code vector indices, and each code vector represents a data point of a specific type or feature.

[0158] Furthermore, the pre-built end-side compression model processes the sample initial visual information based on the initial codebook and outputs the sample compression information, wherein the specific implementation methods of obtaining the sample initial visual information and processing it by the end-side compression model are similar to steps 101 to 103 and are not repeated here; in the training stage of the model, the initial codebook is continuously updated based on the sample compression information output by the initial end-side compression model until the model training is completed, and the updated initial codebook is obtained.

[0159] Furthermore, after obtaining the sample compression information, the following formula is used <1> Calculate the first quantization loss value :

[0160] <1>

[0161] exist <1> In the formula, is the first gradient stopping operator, i.e. , , is the first weighting factor, Represents the initial visual information of the encoder from the input sample The sample compression information of different types extracted from the Characterizes the code vector closest to the sample compression information for verification. It should be noted that the same parameters in the embodiments of the present application have the same meanings, and the repeated parts will not be repeated.

[0162] Furthermore, in order to ensure the stable convergence of the codebook learning process and avoid mode collapse of the codebook, the embodiment of the present application also adopts Normalization encoding strategy. The normalized coding strategy refers to the normalization of the features and discrete codewords extracted by the encoder. Normalization and normalization of the discrete codebook can further improve the stability of training by mapping all latent variables (i.e., features and code vectors extracted by the encoder) onto a hypersphere.

[0163] Further, according to the first quantized loss value The initial codebook is updated and adjusted to obtain the visual codebook. Figure 8 As shown, Figure 8 is an optional model training schematic diagram of the data processing method for machine vision provided in the embodiment of the present application. For different encoding blocks of the multi-level encoder, the formula <1> The method shown obtains corresponding different sample compression information and , and obtain different first quantization loss values ​​respectively, and then determine multiple different sub-codebooks (for example Figure 8 Visual texture codebook and visual semantic codebook in .

[0164] It should be noted that any coding block in the end-side compression model can be based on the formula <1> Training is performed to ensure that each encoder has excellent vector quantization capabilities after training. Then, multiple sub-codebooks are combined to obtain the final visual codebook.

[0165] In some embodiments, Fig. 9 It is an optional edge-cloud decoding model training flow chart of the data processing method for machine vision provided in an embodiment of the present application. The edge-cloud decoding model is trained by following steps 601 to 604:

[0166] Step 601, receiving sample compression information of various types sent from the end-side device, wherein the sample compression information is obtained by the end-side device performing vector quantization processing on sample coding features of various types based on a visual codebook; the sample coding features of various types are obtained by inputting the acquired sample initial visual information into a pre-trained end-side compression model, and performing different types of feature extraction on the sample initial visual information through a multi-level encoder.

[0167] In step 602, the sample compression information of each type is input into the pre-built edge-cloud decoding model, and the sample encoding features represented by the different types of sample compression information are reconstructed through a multi-level decoder to obtain the sample decoding features of the corresponding type at each level.

[0168] Step 603: Fusing sample decoding features of different types to obtain sample target visual information that matches the sample initial visual information.

[0169] Step 604, based on the sample initial visual information and the sample target visual information, the pixel loss value and the perceptual loss value are calculated, and the edge-cloud side parameters of the edge-cloud side decoding model are adjusted according to the pixel loss value and the perceptual loss value to obtain a trained edge-cloud side decoding model.

[0170] Steps 601 to 604 are described in detail below.

[0171] In some embodiments, before the edge-cloud side decoding model is actually applied, the pre-built edge-cloud side decoding model also needs to be trained with sample data so that the trained edge-cloud side decoding model can better transmit and process the input compressed information.

[0172] Furthermore, the pre-built edge-cloud decoding model processes the sample compression information and outputs the sample target visual information. The sample compression information can be obtained from a trained end-side compression model, or obtained from a third-party information source. The specific implementation method of the edge-cloud transmission module processing the obtained sample compression information is similar to steps 401 to 403, which will not be repeated here.

[0173] Furthermore, if Figure 8 As shown, then, the image reconstruction capability of the edge-cloud decoding model is independently trained by the following formula <2> Calculate the pixel loss value :

[0174] <2>

[0175] Through the following formula <3> Calculate the perceptual loss value :

[0176] <3>

[0177] in stands for Multi-Level Encoder; stands for Multi-Level Decoder; Represents shallow encoding / decoding features, Indicates deep encoding / decoding features; Figure 8 In the example, Includes sample texture encoding features and sample texture decoded features , Including sample semantic encoding features And sample semantic decoding features ; represents the image reconstruction module; is the initial visual information of the sample, is the sample target visual information; is a pre-trained deep neural network model for computing perceptual loss. and They are The height and width dimensions, and They are the perceptual features ( and )’s height, width and number of channels.

[0178] Furthermore, using the pixel loss value and the perceived loss value Adjust the edge-cloud side parameters of the edge-cloud side decoding model respectively.

[0179] Furthermore, according to the pixel loss value and the perceived loss value The edge-cloud side parameters of the edge-cloud side decoding model are adjusted. The edge-cloud side parameters include but are not limited to the number of decoder layers, the number of edge-cloud side network neurons, the edge-cloud side network learning rate, edge-cloud side network optimizer parameters, edge-cloud side network regularization parameters, edge-cloud side network batch size, etc. The specific edge-cloud side parameters that need to be adjusted can be set according to actual conditions, and the embodiments of the present application do not limit this.

[0180] Or, by the following formula <4> Calculate the overall reconstruction loss value :

[0181] <4>

[0182] in, is the overall reconstruction loss value including pixel-level loss value and the perceived loss value Two parts, is the preset weight factor. Then, the overall reconstruction loss value The edge-cloud side parameters of the edge-cloud side decoding model are adjusted to obtain a trained edge-cloud side decoding model.

[0183] In some embodiments, Fig.10 This is an optional joint training flowchart of the data processing method for machine vision provided in an embodiment of the present application. The terminal side compression model and the edge cloud side decoding model are obtained by joint training through the following steps 701 to 702, and the steps include:

[0184] Step 701: Based on a preset second gradient stop operator and a second weight factor, a second quantization loss value of a sample encoding feature and a sample decoding feature of the same type is calculated.

[0185] Step 702: jointly adjust the end-side parameters and the edge-cloud side parameters according to the second quantization loss value to obtain a trained end-side compression model and an edge-cloud side decoding model.

[0186] Steps 701 to 702 are described in detail below.

[0187] In some embodiments, the parameters of the end-side compression model and the edge-cloud-side decoding model can also be jointly adjusted. This helps improve the overall performance of the two models while reducing optimization energy consumption, thereby helping the two trained models to work better together in actual applications.

[0188] exist Figure 8 In the example, firstly, based on steps 501 to 505 and steps 601 to 604, relevant sample training data is obtained, and the sample training data includes the client-side compression model based on the sample initial visual information The extracted sample texture encoding features and sample semantic encoding features , so as to calculate the second quantization loss value corresponding to each type of encoding / decoding feature based on the sample training data.

[0189] Furthermore, the second quantized loss value includes a loss value of a shallow encoding / decoding block , and the loss value of deep encoding / decoding blocks , for different levels of encoding / decoding blocks, the calculation of the second quantization loss value is as follows:

[0190] Specifically, through the following formula <5> Calculate the second quantization loss value of the shallow encoding / decoding block :

[0191] <5>

[0192] In-style <5> middle, is the second gradient stopping operator, is the second gradient stopping operator, i.e. , ; Represents the second weight factor corresponding to the shallow encoding / decoding block.

[0193] Furthermore, by the following formula <6> Calculate the second quantization loss value of the deep encoding / decoding block :

[0194] <6>

[0195] in, Represents the second weight factor corresponding to the deep encoding / decoding block, and the formula <5> The same parameters will not be repeated.

[0196] Further, according to the second quantized loss value as well as The end-side parameters and edge-cloud-side parameters are jointly adjusted to train the two models so that in the process of collaboratively processing transmission tasks, the end-side compression model can better output encoding features, while the edge-cloud-side decoding model can better output decoding features.

[0197] Furthermore, the adjusted end-side parameters include but are not limited to the number of encoder layers, the number of end-side network neurons, the end-side network learning rate, the end-side network optimizer parameters, the end-side network regularization parameters, the end-side network batch size, etc. The specific end-side parameters that need to be adjusted can be set according to actual conditions, and the embodiments of the present application do not limit this. The specific content of the adjusted edge-cloud side parameters is similar to step 604 and will not be repeated here.

[0198] In some embodiments, Fig.11 It is another optional joint training flowchart of the data processing method for machine vision provided in an embodiment of the present application. The terminal side compression model and the edge cloud side decoding model are obtained by joint training through the following steps 801 to 802, and the steps also include:

[0199] Step 801, based on the pre-introduced discriminator, discriminate the sample initial visual information to obtain a first discriminant value, and discriminate the sample target visual information to obtain a second discriminant value.

[0200] Step 802: Superimpose the logarithmic processing result of the first discriminant value and the logarithmic processing result of the second discriminant value to obtain an adversarial loss value, and jointly adjust the end-side parameters and the edge-cloud side parameters according to the adversarial loss value to obtain a trained end-side compression model and edge-cloud side decoding model.

[0201] Steps 801 to 802 are described in detail below.

[0202] In some embodiments, Figure 8 As shown, in order to improve the feature reconstruction capability of the transmission device, especially the feature reconstruction capability based on the shallow coding / decoding block, the embodiment of the present application also introduces a patch-based discriminator, so as to impose image reconstruction constraints on the coding / decoding block, especially the shallow coding / decoding block, through adversarial training, thereby improving the image reconstruction quality of the transmission device. <7> Calculate the adversarial loss value :

[0203] <7>

[0204] in, is the discriminator introduced in advance; is the first discriminant value, is the logarithmic processing result of the first discriminant value; is the second discriminant value, is the logarithmic processing result of the second discriminant value.

[0205] Furthermore, the end-side parameters and edge-cloud side parameters are jointly adjusted to train the two models to achieve optimal performance in the process of collaboratively processing transmission tasks. The specific contents of the adjusted end-side parameters and edge-cloud side parameters are similar to steps 701 to 702 and will not be repeated here.

[0206] Furthermore, according to the adversarial loss value The end-side parameters and edge-cloud side parameters are jointly adjusted to train the two models. In the process of collaboratively processing transmission tasks, the end-side compression model can better determine the encoding features that represent the reconstruction, while the edge-cloud side decoding model can better reconstruct the corresponding features.

[0207] In some embodiments, Fig.12 It is another optional joint training flowchart of the data processing method for machine vision provided in an embodiment of the present application. The terminal side compression model and the edge cloud side decoding model are obtained by joint training through the following steps 901 to 903, and the steps also include:

[0208] Step 901, based on the pre-introduced teacher model, feature extraction is performed on the sample initial visual information to obtain the sample prediction decoding feature.

[0209] Step 902, calculate the mean square error value between the sample prediction decoding feature and the sample decoding feature to obtain the distillation loss value.

[0210] Step 903: Jointly adjust the end-side parameters and the edge-cloud side parameters according to the distillation loss value to obtain a trained end-side compression model and an edge-cloud side decoding model.

[0211] Steps 901 to 903 are described in detail below.

[0212] In some embodiments, Figure 8 As shown, in order to improve the feature extraction capability of the transmission device, especially the feature extraction capability based on the deep encoding / decoding block, the embodiment of the present application also introduces a teacher model for distillation training to impose feature distillation constraints on the encoding / decoding blocks, especially the deep encoding / decoding blocks, and then supervise the deep encoding / decoding blocks to learn high-level encoding / decoding features.

[0213] Furthermore, the basic structure of the teacher model usually includes a text encoder and an image encoder, which are used to extract text features and image features respectively, and the teacher model is usually pre-trained on large-scale image-text data, so the teacher model usually has excellent feature extraction capabilities. The end-side compression model and edge-cloud decoding model in the embodiment of the present application will serve as student models, and will be trained for knowledge distillation with the help of the introduced teacher model.

[0214] Furthermore, the teacher model can be a contrastive language-image pre-training model (Contrastive Language-Image Pre-Training, CLIP), a vision-language pre-trained model (Vision-Language Pre-trained Model, VLMo), etc., which can be set according to the actual training situation, and the embodiments of the present application do not limit this.

[0215] Specifically, the embodiment of the present application adopts the feature distillation method, and uses the pre-trained visual features extracted by the teacher model as the feature extraction learning target of the student model in the embodiment of the present application, through the following formula: <8> Calculate the distillation loss value :

[0216] <8>

[0217] in, The sample prediction decoding features extracted by the teacher model from the sample visual information, The sample decoding features obtained by encoding, discretizing, and decoding the sample initial visual information for the edge-side and edge-cloud decoding models; MSE is the mean square error function; stands for Multi-Level Encoder; Stands for Multi-Level Decoder.

[0218] Furthermore, the embodiment of the present application can implement distillation-reconstruction dual-drive constraint training of the device through steps 801 to 802, and steps 901 to 903, so that the encoding / decoding blocks at different levels in the device have the ability to better extract and restore key features from different types of features.

[0219] In some embodiments, Fig.13 This is another optional joint training flowchart of the data processing method for machine vision provided in an embodiment of the present application. The terminal side compression model and the edge cloud side decoding model are obtained by joint training through the following steps 1001 to 1002, and the steps also include:

[0220] Step 1001, according to the preset training weights, superimpose the first quantization loss value, the pixel loss value, the perceptual loss value, the second quantization loss value, the adversarial loss value and the distillation loss value to obtain a total loss value.

[0221] Step 1002: jointly adjust the end-side parameters and the edge-cloud side parameters according to the total loss value to obtain a trained end-side compression model and an edge-cloud side decoding model.

[0222] Steps 1001 to 1002 are described in detail below.

[0223] In some embodiments, in order to improve the transmission device to process different types of features at different levels, the following formula is used: <9> Calculate the total loss value :

[0224] <9>

[0225] in, To minimize The average loss of The loss value of is the target; , , and It is the corresponding weight factor of different items, and each weight factor can be adaptively adjusted according to actual conditions.

[0226] Furthermore, after training, the transmission device proposed in the embodiment of the present application can effectively support the distributed deployment and reasoning of the edge-end cloud, and effectively support various visual perception tasks through lightweight model tuning methods (such as adapter tuning, prompt tuning, etc.). The specific operation steps are as follows:

[0227] ① Distributed deployment of device, edge and cloud: Multi-level encoders are deployed on different perception terminals, multi-level decoders are deployed on the edge or cloud side, and the device, edge and cloud deployments have the same visual codebook;

[0228] ② Device-edge-cloud collaborative reasoning: The device preprocesses the collected initial visual information and extracts continuous coding features through a pre-trained multi-level encoder. Subsequently, the device uses the visual codebook to vector quantize the continuous coding features to obtain compressed information (code vector index), so that efficient communication at extremely low bit rates can be achieved between the device and the cloud or between the device and the edge by transmitting the code vector index.

[0229] For the edge-cloud system, based on the received code vector index, a multi-level decoder is used to recover the visual features that can effectively support downstream tasks. Finally, the reconstructed visual features are input into the task model to perform recognition, segmentation, detection, tracking, understanding and other task processing to achieve end-edge-cloud collaborative reasoning and obtain the final target visual information.

[0230] In order to improve the support capability of the features reconstructed by the hierarchical decoder for downstream tasks, the embodiment of the present application can also use methods such as adapter tuning or prompt tuning to tune the decoder. Specifically, after adding an adapter or prompt token to the decoder, a small amount of data related to the downstream task is used to learn the newly added adapter or prompt token, so that the reconstructed visual features can be better adapted to the downstream tasks.

[0231] like Fig.14 As shown, Fig.14 1101 is an optional flowchart of a data processing device for machine vision provided in an embodiment of the present application. The data processing device for machine vision includes the following modules 1101 to 1104:

[0232] An acquisition module 1101 is used to acquire initial visual information to be transmitted;

[0233] The feature encoding module 1102 is used to input the initial visual information into a pre-trained end-side compression model, and extract different types of features from the initial visual information through a multi-level encoder to obtain corresponding types of encoding features at each level;

[0234] A vector quantization module 1103 is used to perform vector quantization processing on the coding features of each type based on the visual codebook to obtain compression information of the corresponding type;

[0235] The target transmission module 1104 is used to send the corresponding type of compressed information to the edge-cloud side system, so that the edge-cloud side system inputs the compressed information of each type into the pre-trained edge-cloud side decoding model, and reconstructs the encoding features representing the different types of compressed information through a multi-level decoder to obtain the corresponding type of decoding features at each level; and fuses the decoding features of different types to obtain the target visual information that matches the initial visual information.

[0236] The data processing method, device, equipment and medium for machine vision proposed in the present application obtain the initial visual information to be transmitted; input the initial visual information into a pre-trained edge-side compression model, and extract different types of features from the initial visual information through a multi-level encoder to obtain the corresponding type of coding features at each level; then, based on the visual codebook, respectively perform vector quantization processing on the coding features of each type to obtain the corresponding type of compressed information; send the corresponding type of compressed information to the edge-cloud system, so that the edge-cloud system inputs the compressed information of each type into a pre-trained edge-cloud decoding model, and reconstructs the coding features represented by the different types of compressed information through a multi-level decoder to obtain the corresponding type of decoding features at each level; and fuses the decoding features of different types to obtain the target visual information that matches the initial visual information; and, since the present application transmits the compressed information corresponding to the initial visual information determined based on the visual codebook, the amount of information that needs to be transmitted is reduced, so that the edge-cloud system can reconstruct the final target visual information based on the small amount of information transmitted by the edge-side device, thereby improving the transmission efficiency of visual data.‌

[0237] The specific implementation of the data processing device for machine vision is basically the same as the specific implementation of the data processing method for machine vision described above, and will not be repeated here.

[0238] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned data processing method for machine vision when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0239] like Fig.15 As shown, Fig.15 : is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, the electronic device includes:

[0240] The processor 1201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0241] The memory 1202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1202, and the processor 1201 calls and executes the data processing method for machine vision in the embodiment of this application;

[0242] Input / output interface 1203, used to implement information input and output;

[0243] The communication interface 1204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0244] A bus 1205 that transmits information between various components of the device (e.g., the processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204);

[0245] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0246] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data processing method for machine vision is implemented.

[0247] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0248] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0249] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0250] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0251] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0252] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0253] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0254] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0255] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0256] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0257] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0258] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data processing method for machine vision, characterized in that: Applied to the end-side device, the end-side device and the edge-cloud system are both provided with the same visual codebook, and the data processing method for machine vision includes: Acquiring initial visual information to be transmitted; Inputting the initial visual information into a pre-trained end-side compression model, performing different types of feature extraction on the initial visual information through a multi-level encoder, and obtaining corresponding types of encoding features at each level; Based on the visual codebook, vector quantization processing is performed on the coding features of each type to obtain compression information of the corresponding type; Send the compressed information of the corresponding type to the edge-cloud system, so that the edge-cloud system inputs the compressed information of each type into a pre-trained edge-cloud decoding model, and reconstructs the encoding features represented by the compressed information of different types through a multi-level decoder to obtain the decoding features of the corresponding types at each level; and fuses the decoding features of different types to obtain the target visual information matching the initial visual information; The multi-level encoder includes shallow-level coding blocks and deep-level coding blocks; the coding features include texture coding features and semantic coding features; The method of extracting different types of features from the initial visual information through a multi-level encoder to obtain corresponding types of coding features at each level includes: Performing texture coding on the initial visual information based on the shallow coding block to obtain the texture coding feature under the texture type; The texture coding feature is semantically coded based on the deep coding block to obtain the semantic coding feature under the semantic type.

2. The data processing method for machine vision according to claim 1, characterized in that: The visual codebook includes a plurality of code vectors identified by code vector indices; The performing vector quantization processing on the coding features of each type based on the visual codebook to obtain compression information of the corresponding type includes: Performing dimensionality reduction processing on the texture coding feature to obtain a plurality of first basic feature units after the dimensionality reduction of the texture coding feature; Determine a first code vector matching the first basic feature unit from the plurality of code vectors, and use the code vector index corresponding to the first code vector as the compression information under the texture type; Alternatively, a plurality of second basic feature units of the semantic coding feature are determined, a second code vector matching the second basic feature unit is determined from the plurality of code vectors, and the code vector index corresponding to the second code vector is used as the compression information under the semantic type.

3. A data processing method for machine vision, characterized in that: Applied to the edge-cloud system, the same visual codebook is set in both the terminal device and the edge-cloud system, and the data processing method for machine vision includes: receiving compressed information of various types sent by the end-side device, wherein the compressed information is obtained by the end-side device performing vector quantization processing on coding features of various types based on the visual codebook; the coding features of various types are obtained by inputting the acquired initial visual information into a pre-trained end-side compression model and performing different types of feature extraction on the initial visual information through a multi-level encoder; Input the compressed information of each type into a pre-trained edge-cloud decoding model, and reconstruct the encoding features represented by the compressed information of different types through a multi-level decoder to obtain decoding features of the corresponding type at each level; Fusing the decoded features of different types to obtain target visual information matching the initial visual information; The multi-level encoder includes shallow-level coding blocks and deep-level coding blocks; the coding features include texture coding features and semantic coding features; The method of extracting different types of features from the initial visual information through a multi-level encoder to obtain corresponding types of coding features at each level includes: Performing texture coding on the initial visual information based on the shallow coding block to obtain the texture coding feature under the texture type; The texture coding feature is semantically coded based on the deep coding block to obtain the semantic coding feature under the semantic type.

4. The data processing method for machine vision according to claim 3, characterized in that: The visual codebook is obtained by the following steps, comprising: Acquire initial visual information of the sample to be transmitted; Inputting the sample initial visual information into a pre-built end-side compression model, performing different types of feature extraction on the sample initial visual information through a multi-level encoder, and obtaining sample encoding features of corresponding types at each level; Based on a preset initial codebook, vector quantization processing is performed on the sample coding features of each type to obtain sample compression information of the corresponding type; Calculate a first quantization loss value based on the sample compression information, a preset first gradient stop operator and a preset first weight factor; The initial codebook is updated according to the first quantization loss value to obtain the visual codebook.

5. The data processing method for machine vision according to claim 4, characterized in that: The edge-cloud decoding model is trained by the following steps, which include: receiving sample compression information of each type sent by the end-side device, wherein the sample compression information is obtained by the end-side device performing vector quantization processing on the sample coding features of each type based on the visual codebook; the sample coding features of each type are obtained by inputting the acquired sample initial visual information into a pre-trained end-side compression model, and performing different types of feature extraction on the sample initial visual information through a multi-level encoder; Input the sample compression information of each type into the pre-built edge cloud decoding model, reconstruct the sample encoding features represented by the sample compression information of different types through a multi-level decoder, and obtain the sample decoding features of the corresponding type at each level; Fusing the sample decoding features of different types to obtain sample target visual information matching the sample initial visual information; Based on the sample initial visual information and the sample target visual information, a pixel loss value and a perceptual loss value are calculated, and the edge-cloud side parameters of the edge-cloud side decoding model are adjusted according to the pixel loss value and the perceptual loss value to obtain the trained edge-cloud side decoding model.

6. The data processing method for machine vision according to claim 5, characterized in that: The terminal side compression model and the edge cloud side decoding model are both obtained by joint training through the following steps, wherein the steps include: Based on a preset second gradient stop operator and a second weight factor, a second quantization loss value of the sample encoding feature and the sample decoding feature of the same type is calculated; The end-side parameters and the edge-cloud-side parameters are jointly adjusted according to the second quantization loss value to obtain the trained end-side compression model and the edge-cloud-side decoding model.

7. The data processing method for machine vision according to claim 6, characterized in that: The terminal side compression model and the edge cloud side decoding model are both obtained by joint training through the following steps, which also include: Based on the pre-introduced discriminator, the sample initial visual information is discriminated to obtain a first discriminant value, and the sample target visual information is discriminated to obtain a second discriminant value; The logarithmic processing result of the first discriminant value and the logarithmic processing result of the second discriminant value are superimposed to obtain an adversarial loss value, and the end-side parameters and the edge-cloud side parameters are jointly adjusted according to the adversarial loss value to obtain the trained end-side compression model and the edge-cloud side decoding model.

8. The data processing method for machine vision according to claim 7, characterized in that: The terminal side compression model and the edge cloud side decoding model are both obtained by joint training through the following steps, which also include: Based on the pre-introduced teacher model, feature extraction is performed on the initial visual information of the sample to obtain sample prediction decoding features; Calculating a mean square error value between the sample prediction decoding feature and the sample decoding feature to obtain a distillation loss value; The end-side parameters and the edge-cloud-side parameters are jointly adjusted according to the distillation loss value to obtain the trained end-side compression model and the edge-cloud-side decoding model.

9. The data processing method for machine vision according to claim 8, characterized in that: The terminal side compression model and the edge cloud side decoding model are both obtained by joint training through the following steps, which also include: According to a preset training weight, the first quantization loss value, the pixel loss value, the perceptual loss value, the second quantization loss value, the adversarial loss value, and the distillation loss value are superimposed to obtain a total loss value; The end-side parameters and the edge-cloud-side parameters are jointly adjusted according to the total loss value to obtain the trained end-side compression model and the edge-cloud-side decoding model.

10. A data processing device for machine vision, characterized in that: The data processing device for machine vision is provided with a visual codebook, and the device comprises: An acquisition module, used for acquiring initial visual information to be transmitted; A feature encoding module, used to input the initial visual information into a pre-trained end-side compression model, and extract different types of features from the initial visual information through a multi-level encoder to obtain encoding features of corresponding types at each level; A vector quantization module, configured to perform vector quantization processing on the coding features of each type based on the visual codebook to obtain compression information of the corresponding type; A target transmission module is used to send the compressed information of the corresponding type to the edge-cloud side system, so that the edge-cloud side system inputs the compressed information of each type into a pre-trained edge-cloud side decoding model, and reconstructs the encoding features represented by the compressed information of different types through a multi-level decoder to obtain the decoding features of the corresponding types at each level; and fuses the decoding features of different types to obtain the target visual information matching the initial visual information; The multi-level encoder includes shallow-level coding blocks and deep-level coding blocks; the coding features include texture coding features and semantic coding features; The method of extracting different types of features from the initial visual information through a multi-level encoder to obtain corresponding types of coding features at each level includes: Performing texture coding on the initial visual information based on the shallow coding block to obtain the texture coding feature under the texture type; The texture coding feature is semantically coded based on the deep coding block to obtain the semantic coding feature under the semantic type.

11. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the data processing method for machine vision as described in any one of claims 1 to 2 or the data processing method for machine vision as described in any one of claims 3 to 9 when executing the computer program.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the computer program implements the data processing method for machine vision described in any one of claims 1 to 2 or the data processing method for machine vision described in any one of claims 3 to 9.

Images