Machine semantic carrier selection and end-to-end training method

Abstract

The invention provides a machine semantic carrier selection and end-to-end training method. The method comprises the following steps: extracting visual signal representation from visual signal data; extracting the expressed features by using a plurality of feature extraction networks to obtain a plurality of features; selecting one feature meeting a preset limiting condition from the plurality of features as a universal visual semantic carrier, and taking the feature extraction network corresponding to the carrier as a feature generator; a training set, a feature generator and a task network are obtained, each sample in the training set comprises input data and a task label related to a machine vision task, and the input data is represented by a visual signal; using the training set to train a feature generator and a task network, including extracting features of input data from the feature generator to obtain a visual semantic carrier, performing quantization operation to obtain quantized features, and using the task network to obtain a prediction result based on the quantized features, and updating the parameters of the feature generator and the task network according to the difference between the task label and the prediction result.

Description

Technical Field

[0001] The present invention relates to the field of machine semantic compression coding, specifically to the field of machine semantic selection and machine semantic understanding, and more specifically to a method for selecting a machine semantic carrier and end-to-end training. Background Art

[0002] In the era of big data where everything is connected, a large amount of visual data is processed by machines. For example, in smart city scenarios, security monitoring is widely distributed and collects visual data 24 hours a day, resulting in a large amount of new data that needs to be analyzed in a timely manner every day. In the industrial Internet of Things scenario, in order to improve efficiency, the ultra-high frame rate capture technology used will capture hundreds of millions of frames of information per second, which will also generate a large amount of visual data. These scenarios have the requirements of low latency and high precision, which poses a challenge to the compression encoding of visual data.

[0003] After decades of development, the existing standards of traditional hybrid video encoders are based on visual signal statistics and the human visual system. For example, Versatile Video Coding (H.266 / VVC) and the third-generation digital audio and video coding standard (AVS3) can compress the original visual data by about 600 times and ensure that the distortion is almost invisible to the human eye. However, in the face of the exponential growth of visual information data and machine intelligence applications, the signal compression coding paradigm based on human vision can no longer meet the needs. The newly added massive data has prompted machine semantic communication to become the mainstream development technology.

[0004] In addition, the interaction between humans and machines (Human-to-Machine, H2M for short) and machines (Machine-to-Machine, M2M for short) has attracted widespread attention in various IoT scenarios. Therefore, in addition to the increasing visual data in the traditional sense, machine vision data covers a wider range. Machine vision actually includes signals of almost all bands, not just the visible light bands accepted by human vision. For example, in the field of deep space exploration, the Five-hundred-meter Aperture Spherical radio Telescope collects about 3,200 TB of data every day, including non-visible light information in 19 different bands; in the field of ocean exploration, the mantis shrimp bionic camera used also studies the migration habits of organisms by sensing data in non-visible light bands; in addition, infrared cameras, light field cameras and other equipment also generate a large amount of multivariate machine vision data, which brings challenges to compression coding technology.

[0005] In traditional grammatical communication, Shannon defined the grammatical communication problem as "reproducing at one point exactly or approximately a message selected from another point". He believed that the semantic aspects of communication should be considered irrelevant to engineering problems. Guided by Shannon's methods and ideas, most existing communication systems are designed based on indicators centered on throughput, spectrum / energy efficiency, and information reconstruction efficiency. However, this communication concept and technology are difficult to keep up with the exponential growth rate of data traffic. Therefore, the need for efficient communication to support machine intelligence services has triggered a paradigm shift from "semantic neutrality" to "semantic communication".

[0006] From the perspective of machine vision, using visual signals to represent information to train algorithms is feasible, but it is often inefficient because the redundant information caused by the strong correlation in the data features will indirectly lead to a decrease in task accuracy. The application scenarios of machine semantics mainly consider the accuracy of task execution. The encoding process needs to delete data that is redundant to the task. At the same time, it is necessary to ensure the quality of task execution while reducing the data bit rate. However, the current existing methods have the problem that the carrier of machine semantic information is not clear. In addition, the existing technology usually selects machine semantics for a specific machine vision task or model, and selects the features generated in its reasoning process as machine semantics. When there are multiple tasks, it is necessary to add features. As the features increase, more redundant information will be generated, resulting in a large amount of data and reduced accuracy of each task.

[0007] Therefore, when facing multi-task scenarios, existing methods have the problem of unclear and non-standard selection of machine semantic carriers. If each of the multiple tasks obtains feature data in its own processing method, it will lead to a large amount of feature data and more redundant information in the intermediate processing process. Too much redundant information will further lead to the problem of reduced accuracy of each task.

[0008] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0009] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for selecting and end-to-end training a machine semantic carrier.

[0010] The objective of the present invention is achieved through the following technical solutions:

[0011] According to a first aspect of the present invention, a method for selecting a machine semantic carrier is provided, which is used to select a universal visual semantic carrier for one, two or more machine vision tasks in a current application scenario, and the method comprises: S1, extracting the visual signal representation required for the current application scenario from the visual signal data; S2, respectively extracting the features of the visual signal representation using a plurality of feature extraction methods to obtain a plurality of visual semantic features, wherein the plurality of feature extraction methods comprises using a plurality of pre-trained feature extraction networks for feature extraction; S3, selecting a visual semantic feature that meets preset constraints from the plurality of visual semantic features as a universal visual semantic carrier, and using the pre-trained feature extraction network corresponding to the visual semantic carrier as a feature generator, wherein the preset constraints comprise that the accuracy of the visual semantic feature in each of all machine vision tasks is not less than the lower limit of the accuracy requirement of the task.

[0012] In some embodiments of the present invention, in step S3, the preset restriction conditions also include data volume restriction and computational complexity upper limit, and the method of selecting the universal visual semantic carrier includes: selecting visual semantic features that meet the respective lower limit requirements of accuracy for all machine vision tasks from multiple visual semantic features to obtain a feature set; evaluating the data volume of each feature and the computational complexity corresponding to each feature of all visual semantic features of the second feature set, selecting visual semantic features whose computational complexity is less than or equal to the computational complexity upper limit and whose data volume meets the data volume restriction, and using them as universal visual semantic carriers.

[0013] In some embodiments of the present invention, the feature set is obtained as follows:

[0014]

[0015] Where F represents the feature set, represents the selection of visual semantic features that meet the lower accuracy requirements of all machine vision tasks, i represents the type number of machine vision tasks, m represents the total number of machine vision tasks, F i represents the set of visual semantic features that meets the lower limit of accuracy of the i-th task, F represents the visual semantic feature, S represents the set of multiple visual semantic features, Q i represents the lower limit requirement of the accuracy of the i-th task, Φ i (F) represents the accuracy of the features extracted by the pre-trained feature extraction network corresponding to the visual semantic feature F on the i-th task.

[0016] In some embodiments of the present invention, the method of selecting the visual semantic feature whose computational complexity is less than or equal to the computational complexity upper limit and whose data volume satisfies the data volume limit is as follows:

[0017]

[0018] in, It means selecting the visual semantic feature with the smallest data volume and computational complexity less than or equal to the upper limit of computational complexity from the second feature set. represents the amount of data of the visual semantic feature F, st represents the constraint symbol, represents the computational complexity corresponding to the visual semantic feature F, and c represents the upper limit of the computational complexity.

[0019] In some embodiments of the present invention, all machine vision tasks in the current application scenario include classification tasks, target detection tasks, target segmentation tasks, key point detection tasks and face recognition tasks or a combination thereof.

[0020] According to a second aspect of the present invention, there is provided an end-to-end training method for machine semantic understanding, the method comprising:

[0021] According to all machine vision tasks in the current application scenario, a feature generator determined by the method described in any one of the first aspects of the present invention is obtained, and a task network is obtained according to the machine vision task; a training set is obtained, which includes multiple samples, each sample includes input data and a label, the input data is a visual signal representation required for the current application scenario extracted from a visual signal data, and the label is a task label related to the machine vision task; the feature generator and the task network are trained using the training set, including extracting features from the input data by the feature generator to obtain a general visual semantic carrier, and introducing a preset quantization function to perform a quantization operation on the visual semantic carrier to obtain quantized features, obtaining a prediction result related to the machine vision task based on the quantized features through the task network, and updating the parameters of the feature generator and the parameters of the task network with the optimization goal of minimizing the difference between the task label and the prediction result, to obtain a trained feature generator and a trained task network.

[0022] According to the third aspect of the present invention, there is provided a machine semantic understanding system implemented based on the method described in the second aspect of the present invention, the system comprising: a machine vision representation module, used to extract the visual signal representation required for the current application scenario from the visual signal data; a trained feature generator, used to extract the features of the visual signal representation to obtain a general visual semantic carrier; a machine semantic encoding module, used to quantize the visual semantic carrier to obtain the quantized features, and encode the quantized features to obtain a code stream; a machine semantic decoding module, used to decode the code stream to obtain semantic features; and a trained task network, used to obtain prediction results related to the machine vision task in the current application scenario based on the semantic features.

[0023] In some embodiments of the present invention, the machine semantic encoding module includes a first quantization unit and a second quantization unit, wherein the first quantization unit is used to, when the data amount of the visual semantic carrier is less than or equal to the data amount represented by the visual signal, perform a quantization operation on the visual semantic feature using its preset quantization function to obtain a quantized feature; and the second quantization unit is used to, when the data amount of the visual semantic carrier is greater than the data amount represented by the visual signal, obtain a reconstructed semantic feature obtained by reconstructing the visual semantic carrier using a generative network, and perform a quantization operation on the reconstructed semantic feature using its preset quantization function to obtain a quantized feature, wherein the data amount of the reconstructed semantic feature is less than the data amount represented by the visual signal.

[0024] In some embodiments of the present invention, the machine semantic coding module includes a first coding unit and a second coding unit, wherein the first coding unit is used to encode the quantized features by using a Huffman coding method when the number of zero elements in the quantized features is less than a preset threshold value to obtain a feature code stream; the second coding unit is used to encode the quantized features by using a double code table run-length Huffman coding method when the number of zero elements in the quantized features is greater than or equal to the preset threshold value to obtain a feature code stream, wherein the feature code stream includes a first code table for mapping the value of each non-zero element in all non-zero elements and a second code table for mapping the offset of each non-zero element in all non-zero elements and the previous non-zero element.

[0025] According to a fourth aspect of the present invention, there is provided an electronic device, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in any one of the first and second aspects of the present invention by executing the executable instructions.

[0026] Compared with the prior art, the advantages of the present invention are:

[0027] The semantic carrier selection method of the present invention extracts visual signal data through multiple feature extraction methods to obtain multiple features, and selects the features that are most conducive to the accuracy of all tasks in the current application scenario as semantic carriers and corresponding feature generators from the multiple features. It can select universal machine semantic carriers for different machine vision tasks without adding additional feature data according to the increase in tasks, thereby reducing the amount of data. In addition, when facing a multi-task application scenario, the semantic carrier selection method of the present invention is clear, standardized, and highly versatile, and at the same time, it improves the accuracy of each task to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0029] Figure 1A schematic diagram of a flow chart of a method for selecting a machine semantic carrier according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of a process for visual signal representation according to an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of a process of selecting a universal visual semantic carrier based on multiple visual semantic features according to an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of the principle of an end-to-end training process for machine semantic understanding according to an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of the structure of a machine semantic understanding system according to an embodiment of the present invention;

[0034] Figure 6 A schematic diagram of the structural principle of a machine semantic encoding module 300 according to an embodiment of the present invention;

[0035] Figure 7 Schematic diagram of the structure principle of the entropy encoder 305 in the machine semantic encoding module 300 according to an embodiment of the present invention;

[0036] Figure 8 4 is a schematic diagram of the principle of double code table run-length Huffman coding according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] As mentioned in the background technology section, existing methods have the problem of unclear and non-standard selection of machine semantic carriers when facing multi-task scenarios. If each of the multiple tasks obtains feature data in its own processing method, it will lead to a large amount of feature data and more redundant information in the intermediate processing process. Too much redundant information will further lead to the problem of reduced accuracy of each task.

[0039] Based on the above problems, the inventor proposes a method for selecting and end-to-end training a machine semantic carrier. According to one embodiment of the present invention, the present invention provides a method for selecting a machine semantic carrier, which is used to select a general visual semantic carrier for a variety of machine vision tasks in the current application scenario. The selection method includes: extracting a visual signal representation from visual signal data; extracting the features of the representation using a variety of feature extraction networks to obtain a plurality of visual semantic features; selecting a feature that meets the preset restriction conditions as a general visual semantic carrier, and using the feature extraction network corresponding to the carrier as a feature generator, wherein the restriction conditions include that the accuracy of the visual semantic feature in each task among all machine vision tasks is not less than the lower limit requirement of the accuracy of the task. That is, the semantic carrier selection method of the present invention selects the feature that is most conducive to the accuracy of all tasks in the current application scenario from a plurality of features as a semantic carrier and a corresponding feature generator, and can select a general machine semantic carrier for different machine vision tasks, without the need to add additional feature data according to the increase in tasks, thereby reducing the amount of data. In addition, when the semantic carrier selection method of the present invention faces a multi-task application scenario, the semantic carrier selection method is clear, standardized and highly versatile. At the same time, the accuracy of each task is improved to a certain extent.

[0040] According to one embodiment of the present invention, the present invention also provides an end-to-end training method, which is oriented to machine semantic understanding. In the training method, first, a training set, a feature generator and a task network are obtained, and each sample in the training set includes a visual signal representation and a task label; wherein the feature generator selected by the present invention is a feature extraction network that is most conducive to the accuracy of all tasks in the current application scenario, thereby improving the subsequent training efficiency; secondly, when the feature generator and the task network are trained using the training set, the feature generator extracts the features of the visual signal representation to obtain a visual semantic carrier, and the carrier is quantized to obtain the quantized features, thereby reducing the information entropy and making the semantic carrier of the present invention more streamlined; finally, when the feature generator and the task network are trained, the task network obtains a prediction result based on the quantized features, so as to update the parameters of the feature generator and the task network according to the difference between the task label and the prediction result. The task network of the present invention can better understand the streamlined semantic features, reduce the dependence on redundant data, and improve the accuracy of the task. The present invention can improve the network's ability to understand the machine semantic carrier with low information entropy without adding additional training data, which is conducive to providing efficient semantic communication understanding capabilities for future intelligent services and Internet of Things applications, and promoting the development of the Internet of Things.

[0041] In order to better understand the present invention, the selection method of a machine semantic carrier, the end-to-end training method for machine semantic understanding, and the machine semantic understanding system are described in detail below in combination with specific embodiments.

[0042] 1. Selection Method of Machine Semantic Carrier

[0043] According to one embodiment of the present invention, see Figure 1 , which is a flow chart of a method for selecting a machine semantic carrier, the method comprising the following steps S1, S2 and S3:

[0044] In step S1 , a visual signal representation required for the current application scenario is extracted from the visual signal data.

[0045] In a considerable part of computer vision research, in order to achieve good visual task accuracy, stronger visual representation is usually sought. However, for machine vision, since the signal is in an analog signal state, the signal frequency range that the machine can receive is much larger than the visible light range, so that machine vision can obtain various spectrum signals from low frequency to high frequency, and can process the multidimensional world that humans cannot see or cannot see, that is, the scope and meaning covered by machine vision signals and their representations should be wider. Therefore, the present invention first performs signal extraction on the machine vision signal data to obtain the visual signal representation required for the current application scenario to achieve data compression. That is, according to one embodiment of the present invention, a signal representation method selected according to the current application scenario is adopted to extract the visual signal representation required for the current application scenario from the visual signal data obtained by the machine vision.

[0046] According to one embodiment of the present invention, the signal representation method includes multiple methods, for example, including: log-Mel spectrogram (log-mel for short) representation, red-green-blue depth (Red-Green-Blue-Depth, RGBD for short) representation, region of interest (Region of Interest, RoI) representation, HEVC (High Efficiency Video Coding) representation and VVC (Versatile Video Coding) representation, etc.

[0047] According to one embodiment of the present invention, a corresponding signal representation method is selected for each application scenario. For example: if the current application scenario is in the field of autonomous driving, three-dimensional modeling or robot navigation, point cloud representation is used for extraction; if the current application scenario needs to consider the depth dimension based on RGB, RGBD representation is used; if the current application scenario considers the contour and does not consider the color of the picture, grayscale representation is used to obtain a grayscale image. Among them, a better signal representation method can be further selected according to the machine vision task in the current application scenario. For example, the point cloud representation method includes multiple representation methods such as original point cloud, bird's eye view (Bird's Eye View, referred to as BEV), depth map (Depth Map), etc. One of the better representation methods can be selected according to the machine vision task. For example, selecting a bird's eye view representation method is more conducive to the route planning task in autonomous driving.

[0048] According to an embodiment of the present invention, all machine vision tasks in the current application scenario include one, two or more tasks, such as classification tasks, target detection tasks, target segmentation tasks, key point detection tasks, face recognition tasks or a combination thereof. The present invention can further select a better signal representation method according to the machine vision tasks in the current application scenario. For example, see Figure 2 , which is a schematic diagram of the process of visual signal representation. The signal representation method is as follows:

[0049] Y=f * (X), (1)

[0050] Where Y represents the visual signal representation, f * is the optimal representation of the visual signal data X. * The choice of needs to be based on the modality of the current input visual signal data and the task-related side information t to make a decision, that is, to determine the signal representation method suitable for the current task: X,t), where Indicates the selection of the best representation from multiple signal representations. f1 indicates the first optional signal representation, f2 indicates the second optional signal representation, and f n Indicates the nth optional signal representation method. Among them, the task-related side information t refers to the data information required by the task.

[0051] Schematically, before video encoding, adjustments are made according to the task scenario of video encoding. Schematically, in the task of human eye visual encoding, a visual signal representation that can represent the input image or video signal data in YUV format (representing brightness (Y) and chrominance (UV) separately) is used, and then encoding, decoding, and performing task-related predictions are performed, taking into account the information that human vision is not sensitive to chrominance perception. Schematically, the current task is target detection and classification in the visible light range. Since the visual signal data X includes but is not limited to visible light and non-visible light, that is, infrared and ultraviolet rays that are invisible to the human eye are also included, the required visual signal representation Y only needs to include the signal of the visible light part of the visual signal data X.

[0052] In step S2, features represented by the visual signal are extracted respectively using a plurality of feature extraction methods to obtain a plurality of visual semantic features. The plurality of feature extraction methods include using a plurality of pre-trained feature extraction networks to perform feature extraction.

[0053] According to one embodiment of the present invention, even if the original visual signal data is expressed, the obtained visual signal representation is often high-dimensional, which contains a lot of redundant information, resulting in a large amount of computation for subsequent tasks. From the perspective of machine vision, although it is feasible to use the visual signal representation to directly train the task network, it is often inefficient because the redundant information caused by the strong correlation in the data features will indirectly lead to a decrease in task accuracy. Therefore, according to one embodiment of the present invention, a variety of pre-trained feature extraction networks are used to perform feature extraction on the visual information representation respectively to obtain a smaller amount of data and multiple visual semantic features. The present invention further obtains more streamlined, low-dimensional features through the two steps of representation and feature extraction, which can greatly alleviate the computational complexity of compression coding during the execution of machine vision tasks and improve machine computing efficiency. At the same time, redundant information is reduced and the accuracy of subsequent machine vision tasks is improved.

[0054] According to one embodiment of the present invention, a plurality of pre-trained feature extraction networks include but are not limited to AlexNet Convolutional Neural Network (AlexNet), Visual Geometry Group Network with 16 layers (VGG-16), Residual Network with 34 layers (ResNet-34), SqueezeNet Convolutional Neural Network (SqueezeNet), Densely Connected Convolutional Networks (DenseNet), and MobileNetV2 Convolutional Neural Network (Mobilenetv2), or a combination of all of the above networks. In an illustrative manner, if 10 feature extraction methods are used, 10 features (features are in vector or matrix form) will be obtained.

[0055] It should be noted that the above embodiments do not limit the use of a specific network model or a specific feature extraction method to complete a specific task. For a certain visual task, it can be implemented by multiple feature extraction networks and / or multiple feature extraction methods. Even different network layers of the same feature extraction network can derive different features. For example, using the outputs of the 8th, 12th and 16th layers of ResNet-34 as the extracted features, 3 features can be obtained.

[0056] In step S3, a visual semantic feature that meets preset constraints is selected from multiple visual semantic features as a general visual semantic carrier, and a pre-trained feature extraction network corresponding to the visual semantic carrier is used as a feature generator. The preset constraints include that the accuracy of the visual semantic feature in each of all machine vision tasks is not less than the lower limit of the accuracy requirement of the task.

[0057] According to one embodiment of the present invention, the preset restriction conditions also include data volume restriction and computational complexity upper limit. Among them, the data volume restriction is set as follows: under restriction condition one: the accuracy of the visual semantic feature in each of all machine vision tasks is not less than the lower limit of the accuracy requirement of the task, and restriction condition two: the computational complexity corresponding to the visual semantic feature is not greater than the upper limit of the computational complexity, all visual semantic features that meet these two conditions are screened out, and the feature with the smallest data volume is determined from all the visual semantic features. The technical solution of this embodiment can at least achieve the following beneficial technical effects: the present invention selects the feature that is most conducive to the visual task from a variety of features, while being committed to minimizing the data volume and controlling the encoding computational complexity within a reasonable range.

[0058] According to an embodiment of the present invention, the upper limit of the computational complexity can be set to be much smaller than the representation complexity of the visual signal representation. The task execution complexity of the subsequent machine semantic understanding system It should be understood that the specific value of the upper limit of the computational complexity in the present invention is not limited and can be set according to demand, that is, the upper limit of the computational complexity c is set to meet the following conditions, and the actual upper limit value is set according to demand:

[0059]

[0060] Among them, for the representation complexity and task execution complexity To determine the computational complexity, the most intuitive evaluation index can be used, that is, the task execution time on a computing device with the same resource configuration, or the number of floating point operations (FLOPs) and multiply-accumulate operations (MACs) and other complexity evaluation indicators.

[0061] According to one embodiment of the present invention, see Figure 3 , which is a schematic diagram of the process of selecting a universal visual semantic carrier based on multiple visual semantic features in step S3. The feature extraction method of each visual semantic feature is represented by F=g(Y), where F represents the visual semantic feature obtained by extracting the feature of the visual signal representation Y, and g(·) represents the feature extraction method. Figure 3 As shown in , after extracting visual semantic features through multiple feature extraction methods, the selection method of the general visual semantic carrier includes the following steps S31 and S32:

[0062] Step S31 : Selecting visual semantic features that meet the respective lower limit requirements of accuracy for all machine vision tasks from a plurality of visual semantic features to obtain a feature set.

[0063] According to one embodiment of the present invention, step S31 is semantic carrier selection decision one, which includes: extracting visual semantic features that meet the lower limit accuracy requirements of each machine vision task from multiple visual semantic features to obtain a first feature set; taking the union of the first feature sets of all machine vision tasks to obtain a second feature set that meets the lower limit accuracy requirements of all machine vision tasks.

[0064] The second feature set is obtained as follows:

[0065]

[0066] Where F represents the second feature set, represents the union of the first feature sets of all machine vision tasks, i represents the type number of the machine vision task, m represents the total number of machine vision task types, F i represents the first feature set corresponding to the i-th task, F represents the visual semantic feature, S represents the set of multiple visual semantic features, Q i represents the lower limit requirement of the accuracy of the i-th task, Φ i (F) represents the accuracy of the features extracted by the pre-trained feature extraction network corresponding to the visual semantic feature F on the i-th task.

[0067] Step S32, evaluate the data volume of each feature and the computational complexity corresponding to each feature in all visual semantic features of the second feature set, select visual semantic features whose computational complexity is less than or equal to the computational complexity upper limit and whose data volume meets the data volume limit, and use them as general visual semantic carriers.

[0068] According to an embodiment of the present invention, step S32 is semantic carrier selection decision 2: the method of selecting visual semantic features whose computational complexity is less than or equal to the computational complexity upper limit and whose data volume meets the data volume limit is as follows:

[0069]

[0070] in, It means selecting the visual semantic feature with the smallest data volume and computational complexity less than or equal to the upper limit of computational complexity from the second feature set. represents the amount of data of the visual semantic feature F, st represents the constraint symbol, represents the computational complexity corresponding to the visual semantic feature F, and c represents the upper limit of the computational complexity.

[0071] According to one embodiment of the present invention, the data amount of each feature in all the visual semantic features of the second feature set can be evaluated by bit rate, and the value of the bit rate can be understood as the size of each visual semantic feature after encoding and compression. The computational complexity corresponding to each feature of the second feature set can be evaluated by encoding it, and the computational complexity determined by the encoding process is used as the computational complexity corresponding to the feature. Therefore, a visual semantic feature with a computational complexity less than or equal to the upper limit of the computational complexity and the smallest bit rate is selected from the second feature set as a universal visual semantic carrier. It should be understood that this is only for illustration, and the data amount of each feature and the computational complexity corresponding to each feature can be determined and expressed in other ways, such as the data amount is expressed by its memory usage, and the computational complexity corresponding to each feature can also be expressed by the computational complexity in the feature extraction process, and the present invention is not limited to this.

[0072] 2. End-to-end training method for machine semantic understanding

[0073] The existing technology focuses on using fixed pre-trained deep learning models such as Faster-RCNN or YOLO series to perform visual tasks, but still relies on machine semantic information with large information entropy to output more accurate prediction results. As a man-made product, the key feature of machine vision is that it is easy to guide and train, and its nature is plastic and changeable. Therefore, the present invention uses an end-to-end training method to guide the machine to better accept the ability of simplified semantics, so as to achieve more accurate prediction results based on low information entropy machine semantic information.

[0074] According to an embodiment of the present invention, an end-to-end training method for machine semantic understanding is provided. Figure 4 , which is a schematic diagram of the principle of the end-to-end training process for machine semantic understanding. During the training process, the source-side neural network and the sink-side neural network of the end-to-end neural network need to be trained collaboratively using the training set, that is, the parameters of the source-side neural network and the sink-side neural network are reversely updated through the difference between the predicted results output by the sink-side neural network and the actual results. Among them, the source-side neural network corresponds to the feature generator, and the sink-side neural network corresponds to the task network. Therefore, this aspect proposes to use the reconstructed machine semantics in the end-to-end training process. (That is, the quantized semantic features obtained by quantizing the general visual semantic carrier F output by the feature generator ) is replaced with the general visual semantic carrier F output by the feature generator in the original end-to-end neural network, so that the entire network can better adapt to the quantized semantic features That is, the activation function behind the feature generator is replaced by the quantization function to achieve quantization operation on the general visual semantic carrier, so that the network model can learn to understand the concise semantics with low information entropy. The network model trained in this way can better adapt to the concise machine semantics.

[0075] According to one embodiment of the present invention, an end-to-end training method for machine semantic understanding includes the following steps A1, A2 and A3:

[0076] Step A1: According to all machine vision tasks in the current application scenario, a feature generator determined by the selection method described in the above embodiment is obtained, and a task network is obtained according to the machine vision task.

[0077] According to one embodiment of the present invention, the task network is determined by all machine vision tasks in the current application scenario. If the current application scenario has only one task, the task network only contains one subtask network related to the task. If the current application scenario has multiple tasks, each task corresponds to a subtask network, and the task network contains multiple subtask networks related to each task. For example, if the current application scenario has only two tasks, namely, the target detection task and the classification task, a region candidate network (RPN) is added as a subtask network for the target detection task, and a fully connected network and a softmax regression function are added for the classification task, and the fully connected network and the softmax regression function are used as another subtask network.

[0078] Step A2: Obtain a training set, which includes multiple samples, each sample includes input data and a label, the input data is a visual signal representation required for the current application scenario extracted from a visual signal data, and the label is a task label related to the machine vision task.

[0079] According to one embodiment of the present invention, the method of extracting the visual signal representation is the same as step S1 of the above embodiment, which will not be repeated here. The setting of the label is related to the task. For example, if it is a classification task, the label is the corresponding category label, and if it is a target detection task, the label is the information of the bounding box where the target is located.

[0080] Step A3: training the feature generator and the task network using the training set, including extracting features from the input data by the feature generator to obtain a general visual semantic carrier, and introducing a preset quantization function to quantize the visual semantic carrier to obtain a quantized feature vector. The task network is used to obtain prediction results related to the machine vision task based on the quantized features. The parameters of the feature generator and the task network are updated with the optimization goal of minimizing the difference between the task label and the prediction result to obtain a trained feature generator and a trained task network.

[0081] According to an embodiment of the present invention, the preset quantization function adopts a conventional quantization function. For example, a quantization function in the following form may be adopted:

[0082]

[0083] in, represents the preset quantization function, x represents the general visual semantic carrier, x is in the form of a vector or matrix, step is the quantization step size, Indicates rounding up. step is a non-zero real number. The larger the step, the greater the degree of quantization. Among them, step needs to be flexibly set according to the value range of the value in x. For example, when x is a vector, the value range of each element x in the vector is x∈[-10,10]. At this time, step can be set to 1, 2, 3 or 4, which generally does not exceed the absolute value of the value range of the element. And It is more inclined to compress feature data, similar to the activation function ReLU in convolutional neural networks. The negative feature elements with low task relevance are set to 0. Therefore, the quantization function proposed in the present invention not only supports the conversion of visual semantic carriers into integers, but also supports the conversion of positive feature elements into integers. It is quantized according to the quantization step size, and finally the quantized feature F′ is obtained to achieve data compression. That is, the activation function behind the feature extractor is replaced by the quantization function. The model trained in this way can better adapt to the simplified machine semantics.

[0084] According to one embodiment of the present invention, before executing the process of step A3 above, the machine semantics F in the original end-to-end network is used to train the feature generator and the task network until the feature generator and the task network converge. That is, the feature generator extracts features from the input data to obtain a general visual semantic carrier, and the task network directly obtains the prediction results related to the machine vision task based on the general visual semantic carrier. The parameters of the feature generator and the task network are updated with the optimization goal of minimizing the difference between the task label and the prediction result, until the feature generator and the task network converge, and then the training process of this part is stopped. Then use the quantized feature Participating in training, that is, executing the training process of step A3, enables the training method of the present invention to achieve better convergence without the need for additional training data, and at the same time, obtain more accurate prediction results.

[0085] 3. Machine Semantic Understanding System

[0086] According to one embodiment of the present invention, a machine semantic understanding system is provided based on the method described in the above embodiment. Figure 5, which is a schematic diagram of the structure of the machine semantic understanding system. The system includes: a machine vision representation module 100, which is used to extract the visual signal representation Y required for the current application scenario from the visual signal data X; a trained feature generator 200, which is used to extract the features of the visual signal representation to obtain a universal visual semantic carrier F; a machine semantic encoding module 300, which is used to quantize the visual semantic carrier to obtain the quantized feature The quantized features are encoded to obtain a code stream; the machine semantic decoding module 400 is used to decode the code stream to obtain semantic features; the trained task network 500 is used to obtain prediction results related to the machine vision task in the current application scenario based on the semantic features.

[0087] According to one embodiment of the present invention, see Figure 6 , which is a schematic diagram of the structural principle of the machine semantic coding module 300. The machine semantic coding module includes a data volume judgment module 301, a generation network 302, a first quantization unit 303, a second quantization unit 304 and an entropy coding module 305. Among them, 301 is used to judge the data volume of the visual semantic carrier and the data volume between the visual signal representation. The first quantization unit 303 is used to quantize the visual semantic features using its preset quantization function when the data volume of the visual semantic carrier is less than or equal to the data volume represented by the visual signal, obtain the quantized features, and input them into the entropy encoder 305 for encoding to obtain a code stream. The second quantization unit 304 is used to obtain the reconstructed semantic features obtained by reconstructing the visual semantic carrier using the generation network 302 when the data volume of the visual semantic carrier is greater than the data volume represented by the visual signal, quantize the reconstructed semantic features using its preset quantization function, obtain the quantized features, and input them into the entropy encoder 305 for encoding to obtain a code stream, wherein the data volume of the reconstructed semantic features is less than the data volume represented by the visual signal. The technical solution of the above embodiment can at least achieve the following beneficial technical effects: after the machine semantic coding module of the present invention determines the data amount by 301, it is necessary to reduce the data amount of the visual semantic carrier in one of the above two ways to achieve the maximum compression and reduction of the data amount of the visual semantic carrier F, so that the information entropy of the visual semantic carrier is significantly reduced and then enters the entropy encoder.

[0088] According to an embodiment of the present invention, the preset quantization function of the first quantization unit 303 adopts a conventional quantization function. For example, the formula (5) in the above embodiment may be used: The same quantization formula is used for quantization operation. The preset quantization function of the second quantization unit 304 also adopts a conventional quantization function. For example, the preset quantization function of the second quantization unit 304 is It can take the following forms:

[0089]

[0090] Among them, P F =Gen(F),P F represents the reconstructed semantic features obtained by reconstructing the visual semantic carrier using the generation network 302, and Gen(·) represents the generation network. The network model that can be used by the generation network includes a restricted Boltzmann machine, a generative adversarial network, a variational autoencoder or a self-organizing map, etc., and the present invention is not limited to this.

[0091] According to one embodiment of the present invention, for the case where the amount of data of the visual semantic carrier is greater than the amount of data represented by the visual signal. Taking the task of feature extraction and image target detection based on a convolutional neural network as an example, although the visual semantic carrier (the feature map in this case) generated during the convolution process will have a length and width scaling of 1 / 8 to 1 / 2 compared to the original input visual signal representation Y (the digital image in this case), it will have 5 to 40 times more channels than the visual signal representation Y, resulting in the amount of data of the feature map being much larger than the amount of data of the visual signal representation Y. In addition, compared to the uint8-bit type data of the visual signal representation Y, most convolutional neural networks use high-bit width data such as flop32 or flop16 to pursue higher accuracy. Therefore, when the machine semantic carrier is a feature map, the amount of data will increase greatly. At this time, it is necessary to reconstruct the visual semantic carrier based on F through the generative network 302 to obtain a reconstructed semantic feature, that is, a potential code P F The characteristics of the potential code include: 1) the amount of feature data is reduced, and 2) it can be approximately reconstructed as a visual signal representation. Therefore, the present invention can effectively realize the compression coding of the data volume, and at the same time, improve the semantic understanding ability and adaptability of the task network to low information entropy.

[0092] According to one embodiment of the present invention, see Figure 7, which is a schematic diagram of the structural principle of the entropy encoder 305 in the machine semantic encoding module 300. The entropy encoder 305 in the machine semantic encoding module 300 includes a data distribution analysis module 3051, a first encoding unit 3052 and a second encoding unit 3053. Among them, the entropy encoding module 304 is used to take the quantized features as the data to be encoded, and the elements of the data to be encoded should all be integers. First, the number of zero elements in the data to be encoded is analyzed by the data distribution analysis module 3051. When the number of zero elements in the quantized features is less than the preset threshold, the first encoding unit 3052 is used to encode the quantized features using the Huffman coding method to obtain a feature code stream. When the number of zero elements in the quantized features is greater than or equal to the preset threshold, the second encoding unit 3053 is used to encode the quantized features using the double code table run Huffman coding method to obtain a feature code stream, wherein the feature code stream includes a first code table for mapping the value of each non-zero element in all non-zero elements and a second code table for mapping the offset of each non-zero element in all non-zero elements to the previous non-zero element. The preset threshold value may be set to 50%, 60% or 65%, etc., and the present invention is not limited thereto. Taking the preset threshold value of 50% as an example, when the number of 0 elements is less than 50%, the first encoding unit 3052 is used to directly use Huffman coding on the data to be coded; but when the number of 0 elements in the data to be coded is ≥50%, the second encoding unit 3053 is used for coding.

[0093] According to an embodiment of the present invention, when performing double code table run-length Huffman coding, first, the data to be coded can be regarded as an alternating arrangement of non-zero element values ​​and offsets, and this arrangement is regarded as a formatted data, that is, a non-zero element value code table and an offset code table are constructed. The two code tables are formed by generating the frequency of occurrence of each non-zero element / offset length through Huffman coding to obtain the first code table and the second code table respectively. Figure 8 , which is a schematic diagram of the principle of double code table run-length Huffman coding. The form of the data to be encoded in the figure is: [1,0,0,0,0,0,1,0,0,0,1,3,0,0,0,0,0,…2,0], the constructed non-zero element value code table is: [1,1,1,3,…2], and the end offset does not need to be encoded, so the constructed offset code table is: [0,5,3,0,n]. Therefore, two code tables, non-zero element value code table B and offset code table, are formed respectively. The non-zero element value code table is used as the first code table to map non-zero element values, and the offset code table is used as the second code table to map offsets, thereby forming a formatted code stream. When the decoding module 400 decodes, the two code tables are used alternately (the second code table is used first), and the quantized features can be restored one by one.

[0094] To verify the beneficial effects of the present invention, the inventors conducted the following experiments:

[0095] The following three datasets were selected:

[0096] ①Pubfig83+LFW dataset: It is a combination of two face datasets, PubFig83 dataset and LFW dataset, containing a total of 24,618 face images. Among them, 83 individuals are from PubFig83 dataset, 2 / 3 (8,720 images) are used for training, and 1 / 3 (4,282 images) are used for testing. Specifically, 12,066 faces in the LFW dataset are used as interference to simulate the recognition of non-face subjects in uncontrolled galleries in the real world.

[0097] ②FGVC-Aircraft dataset: This benchmark dataset focuses on fine-grained visual classification of aircraft. It contains 10,200 aircraft images, 100 images of each of 102 aircraft models, and most of them are aircraft. Each image is annotated with the corresponding aircraft model label. Among them, 6,800 images are selected for training and verification, and the remaining 3,400 images are used for testing.

[0098] ③Caltech101 dataset: It consists of 101 different object categories, each category has 40 to 800 images, including indoor scene images, outdoor scene images, and some computer-generated images. The resolution of the image is 300×200 pixels, and each image is labeled with the corresponding category. Among them, 70% of the images are randomly selected as the training set, and the remaining 30% of the data is used as the test set.

[0099] Three typical models were selected as end-to-end neural networks, namely: VGG-16, ResNet-34 and Mobilenetv2. The convolutional layer part was used as the feature generator, and the part after the feature generator was used as the task network.

[0100] (2) Training using the above dataset and model

[0101] For the classification task, the last convolution layer of each feature generator is selected as the machine vision semantic carrier. And in the training process, it is trained using quantization functions with different step lengths. Specifically, first, in the end-to-end training process, the activation function behind each feature extraction network is replaced by the quantization function. The model trained in this way can better adapt to the simplified machine semantics, that is, the model learns to use the simplified semantics. Secondly, the quantization function Quantization uses formula (5) Try adjusting the length of the quantization step. The quantization step sizes are defined as 2, 4, and 8 in the experiment. As a control, the original model in an unquantized state is also involved in the experiment. After 200 training iterations for each model, a trained model is obtained. Finally, the test set and the trained model are used for prediction. The information entropy of the quantized semantic feature F corresponding to each sample is calculated and averaged to obtain the average information entropy H(F). ↓ , and statistical classification accuracy Acc. ↑ Finally, the experimental data in Table 1 are obtained:

[0102] Table 1: Experimental data

[0103]

[0104]

[0105] Evaluation and measurement are performed according to Table 1 above:

[0106] Since information entropy H(F) ↓ The lower it is, the less redundant information is contained in the data. Acc. ↑ It represents the task accuracy, while it represents the classification accuracy in the current task scenario experiment. According to Table 1 above, the selection method and end-to-end training method of the present invention enable data to be compressed more efficiently without losing information, thereby reducing the requirements for storage space and transmission bandwidth. At the same time, while significantly reducing the information entropy of machine semantics, a fairly high task accuracy rate is maintained.

[0107] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0108] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0109] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0110] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for selecting a machine semantic carrier, for selecting a general visual semantic carrier for one, two or more machine vision tasks in a current application scenario, the method comprising: S1, extracting the visual signal representation required for the current application scenario from the visual signal data; S2, using a plurality of feature extraction methods to respectively extract features represented by the visual signal to obtain a plurality of visual semantic features, wherein the plurality of feature extraction methods include using a plurality of pre-trained feature extraction networks to perform feature extraction; S3. Select a visual semantic feature that meets preset constraints from multiple visual semantic features as a general visual semantic carrier, and use a pre-trained feature extraction network corresponding to the visual semantic carrier as a feature generator. The preset constraints include that the accuracy of the visual semantic feature in each of all machine vision tasks is not less than the lower limit of the accuracy requirement of the task.

2. The method according to claim 1, characterized in that In step S3, the preset restriction conditions also include data volume restriction and computational complexity upper limit, and the selection method of the general visual semantic carrier includes: Selecting visual semantic features that meet the lower accuracy requirements of all machine vision tasks from multiple visual semantic features to obtain a feature set; Evaluate the data volume of each feature and the computational complexity corresponding to each feature in all visual semantic features of the second feature set, select visual semantic features whose computational complexity is less than or equal to the computational complexity upper limit and whose data volume meets the data volume limit, and use them as universal visual semantic carriers.

3. The method according to claim 2, characterized in that The feature set is obtained as follows: Where F represents the feature set, represents the selection of visual semantic features that meet the lower accuracy requirements of all machine vision tasks, i represents the type number of machine vision tasks, m represents the total number of machine vision tasks, F i represents the set of visual semantic features that meets the lower limit of accuracy of the i-th task, F represents the visual semantic feature, S represents the set of multiple visual semantic features, Q i represents the lower limit requirement of the accuracy of the i-th task, Φ i (F) represents the accuracy of the features extracted by the pre-trained feature extraction network corresponding to the visual semantic feature F on the i-th task.

4. The method according to claim 2, characterized in that: The method of selecting the visual semantic features whose computational complexity is less than or equal to the computational complexity upper limit and whose data volume meets the data volume limit is as follows: in, It means selecting the visual semantic feature with the smallest data volume and computational complexity less than or equal to the upper limit of computational complexity from the second feature set. represents the amount of data of the visual semantic feature F, st represents the constraint symbol, represents the computational complexity corresponding to the visual semantic feature F, and c represents the upper limit of the computational complexity.

5. The method according to any one of claims 1 to 4, characterized in that: All machine vision tasks in the current application scenarios include classification tasks, object detection tasks, object segmentation tasks, key point detection tasks and face recognition tasks or their combinations.

6. An end-to-end training method for machine semantic understanding, characterized in that: The method comprises: According to all machine vision tasks in the current application scenario, obtain a feature generator determined by the method described in any one of claims 1 to 5, and obtain a task network according to the machine vision tasks; Obtain a training set, which includes multiple samples, each sample includes input data and a label, the input data is a visual signal representation required for a current application scenario extracted from a visual signal data, and the label is a task label related to the machine vision task; The feature generator and the task network are trained using the training set, including extracting features from the input data through the feature generator to obtain a general visual semantic carrier, and introducing a preset quantization function to perform a quantization operation on the visual semantic carrier to obtain quantized features, obtaining prediction results related to the machine vision task based on the quantized features through the task network, and updating the parameters of the feature generator and the parameters of the task network with the optimization goal of minimizing the difference between the task label and the prediction result, thereby obtaining a trained feature generator and a trained task network.

7. A machine semantic understanding system implemented based on the method of claim 6, characterized in that: The system comprises: A machine vision representation module, used to extract the visual signal representation required by the current application scenario from the visual signal data; A trained feature generator is used to extract features represented by visual signals to obtain a general visual semantic carrier; A machine semantic encoding module, used for performing a quantization operation on the visual semantic carrier to obtain quantized features, and encoding the quantized features to obtain a code stream; The machine semantic decoding module is used to decode the code stream to obtain semantic features; The trained task network is used to obtain prediction results related to the machine vision task in the current application scenario based on semantic features.

8. The system according to claim 7, characterized in that The machine semantic encoding module includes a first quantization unit and a second quantization unit, wherein: A first quantization unit is used to perform a quantization operation on the visual semantic feature using a preset quantization function to obtain a quantized feature when the data amount of the visual semantic carrier is less than or equal to the data amount represented by the visual signal; The second quantization unit is used to obtain a reconstructed semantic feature obtained by reconstructing the visual semantic carrier using a generative network when the data amount of the visual semantic carrier is greater than the data amount represented by the visual signal, and to perform a quantization operation on the reconstructed semantic feature using its preset quantization function to obtain a quantized feature, wherein the data amount of the reconstructed semantic feature is less than the data amount represented by the visual signal.

9. The system according to claim 7 or 8, characterized in that: The machine semantic encoding module includes a first encoding unit and a second encoding unit, wherein: A first encoding unit, configured to encode the quantized features by using a Huffman encoding method to obtain a feature code stream when the number of zero elements in the quantized features is less than a preset threshold; The second encoding unit is used to encode the quantized features by using a double-code table run-length Huffman coding method to obtain a feature code stream when the number of zero elements in the quantized features is greater than or equal to a preset threshold, wherein the feature code stream includes a first code table for mapping the value of each non-zero element in all non-zero elements and a second code table for mapping the offset of each non-zero element in all non-zero elements to the previous non-zero element.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 6.

11. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 6 by executing the executable instructions.

Images