Intelligent edge computing methods, systems, terminals and media for cross-scenario deployment

Abstract

The intelligent edge computing method, system, terminal and medium for cross-scenario deployment provided by the present invention specifically relate to the field of information and communication technology, and the scheme includes: using several IoT devices to simultaneously collect image frames of the same object in the source domain or target domain at different perspectives to obtain source domain data and target domain data; based on the source domain data and the target domain data, using unsupervised domain adaptation algorithm and knowledge extraction algorithm to fine-tune the semantic edge reasoning system to obtain an optimized semantic edge reasoning system, and the semantic edge reasoning system is built based on IoT devices and at least one edge server; using the optimized semantic edge reasoning system to perform edge computing on the target domain data to obtain prediction results. The scheme can adaptively handle changes in data distribution and wireless channel signal-to-noise ratio, and does not need to rely on the labeled data of the target domain, and can achieve rapid and economical large-scale deployment in dynamic application scenarios.

Description

Technical Field

[0001] The present invention relates to the field of information and communication technologies, and in particular to an intelligent edge computing method, system, terminal and medium for cross-scenario deployment. Background Art

[0002] At present, intelligent IoT systems based on deep neural network deployment can be applied to perform complex reasoning tasks. However, semantic communication is easily affected by dynamic environments. When the environment in which wireless IoT devices are located changes, the distribution of sensor data collected by wireless IoT devices and the signal-to-noise ratio of the wireless channel will change. Therefore, there is an urgent need for a semantic edge reasoning system that can take into account changes in the signal-to-noise ratio of the wireless channel and the deployment location of wireless IoT devices.

[0003] In the existing semantic edge reasoning system, when semantic communication is affected by the environment and the distribution of sensor data and the signal-to-noise ratio of the wireless channel change, in order to ensure the reasoning accuracy of the system, the wireless IoT device is usually required to re-collect data samples and annotate them, and re-end-to-end training is performed between the wireless IoT device and the mobile edge computing server; or, multiple deep neural network models that are adapted to different channel environments are pre-trained under offline conditions, and the pre-trained model that best meets the real-time signal-to-noise ratio is used in actual applications. However, the acquisition of samples and the training of models are very time-consuming and labor-intensive, resulting in a high system deployment cost. It can be seen that the semantic edge reasoning system in the prior art cannot well ensure the reasoning accuracy of the system when the distribution of sensor data and the signal-to-noise ratio of the wireless channel change, and cannot be deployed on a large scale when the number of samples is small. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an intelligent edge computing method, system, terminal and medium for cross-scenario deployment, aiming to solve the problems in the prior art that the system's reasoning accuracy cannot be well guaranteed when the distribution of sensor data and the signal-to-noise ratio of the wireless channel change, and large-scale deployment cannot be carried out when the number of samples is small.

[0005] In order to achieve the above object, the first aspect of the present invention provides an intelligent edge computing method for cross-scenario deployment, including:

[0006] Use several IoT devices to simultaneously collect image frames of the same object in the source domain or target domain at different perspectives to obtain source domain data and target domain data;

[0007] Based on the source domain data and the target domain data, fine-tuning the semantic edge reasoning system using an unsupervised domain adaptation algorithm and a knowledge extraction algorithm to obtain an optimized semantic edge reasoning system, wherein the semantic edge reasoning system is constructed based on the Internet of Things device and at least one edge server;

[0008] The optimized semantic edge reasoning system is used to perform edge computing on the target domain data to obtain a prediction result.

[0009] Optionally, the fine-tuning of the semantic edge reasoning system based on the source domain data and the target domain data by using an unsupervised domain adaptation algorithm and a knowledge extraction algorithm to obtain an optimized semantic edge reasoning system includes:

[0010] Using an unsupervised domain adaptation algorithm to align feature distributions of the source domain data and the target domain data, and obtaining updated parameters corresponding to the semantic edge reasoning system;

[0011] Based on the updated parameters, the semantic edge reasoning system is fine-tuned according to the channel conditions of the target domain using a knowledge extraction algorithm to obtain an optimized semantic edge reasoning system.

[0012] Optionally, the using an unsupervised domain adaptation algorithm to distribute align the source domain data with the target domain data to obtain updated parameters corresponding to the semantic edge reasoning system includes:

[0013] Extracting semantic features and performing compression coding on the source domain data and the target domain data to obtain a source domain semantic coding sequence and a target semantic coding sequence;

[0014] Calculating a domain adaptation loss between the source domain semantic coding sequence and the target semantic coding sequence using an unsupervised domain adaptation algorithm;

[0015] Based on the domain adaptation loss, feature distributions of the source domain data and the target domain data are aligned to obtain updated parameters corresponding to the semantic edge reasoning system.

[0016] Optionally, the distribution alignment of the source domain data and the target domain data based on the domain adaptation loss to obtain updated parameters corresponding to the semantic edge reasoning system includes:

[0017] Decoding the source domain semantic coding sequence to obtain a prediction result of the source domain data;

[0018] Calculating the source domain cross entropy loss based on the prediction result of the source domain data and the corresponding true value;

[0019] Obtaining a total domain adaptation loss based on the source domain cross entropy loss and the domain adaptation loss;

[0020] Based on the total domain adaptation loss, feature distributions of the source domain data and the target domain data are aligned to obtain updated parameters corresponding to the semantic edge reasoning system.

[0021] Optionally, the fine-tuning of the semantic edge reasoning system according to the channel condition of the target domain by using a knowledge extraction algorithm based on the updated parameters to obtain an optimized semantic edge reasoning system includes:

[0022] Based on the deep neural network model corresponding to the updated parameters, a student model with the same structure is constructed;

[0023] Based on the deep neural network model corresponding to the updated parameters, the source domain data and the target domain data, using a knowledge extraction algorithm to fine-tune the student model according to the channel conditions of the target domain to obtain an optimized student model;

[0024] Based on the optimized student model, an optimized semantic edge reasoning system is obtained.

[0025] Optionally, the deep neural network model corresponding to the updated parameters, the source domain data, and the target domain data, using a knowledge extraction algorithm to fine-tune the student model according to the channel condition of the target domain to obtain an optimized student model, includes:

[0026] Calculating a first prediction result of the target domain data based on the deep neural network model corresponding to the updated parameters;

[0027] Filtering the target domain data based on the confidence of the first prediction result to obtain filtered target domain data;

[0028] Based on the student model, calculating a second prediction result of the filtered target domain data;

[0029] Based on the first prediction result and the second prediction result, the student model is fine-tuned according to the channel condition of the target domain to obtain an optimized student model.

[0030] Optionally, the fine-tuning the student model according to the channel condition of the target domain based on the first prediction result and the second prediction result to obtain an optimized student model includes:

[0031] Calculating the cross entropy between the first prediction result and the second prediction result to obtain a knowledge extraction loss;

[0032] Obtaining a total knowledge extraction loss based on the source domain cross entropy loss, the domain adaptation loss, and the knowledge distillation loss;

[0033] Based on the total knowledge extraction loss, the student model is fine-tuned according to the channel condition of the target domain to obtain an optimized student model.

[0034] A second aspect of the present invention provides an intelligent edge computing system for cross-scenario deployment, the system comprising:

[0035] A data acquisition module, used to use a number of IoT devices to simultaneously acquire image frames of the same object in a source domain or a target domain at different viewing angles to obtain source domain data and target domain data;

[0036] A system optimization module, configured to fine-tune the semantic edge reasoning system based on the source domain data and the target domain data using an unsupervised domain adaptation algorithm and a knowledge extraction algorithm to obtain an optimized semantic edge reasoning system, wherein the semantic edge reasoning system is constructed based on the Internet of Things device and at least one edge server;

[0037] A prediction module is used to perform edge computing on the target domain data using the optimized semantic edge reasoning system to obtain a prediction result.

[0038] The third aspect of the present invention provides a terminal, which includes a memory for storing executable instructions; and a processor for calling and running the executable instructions in the memory to execute any step of the above-mentioned intelligent edge computing method for cross-scenario deployment.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium, which stores program instructions. When the program instructions are executed by a processor, any step of the above-mentioned intelligent edge computing method for cross-scenario deployment is implemented.

[0040] Compared with the prior art, the beneficial effects of this solution are as follows:

[0041] (1) The semantic edge reasoning system designed in this application uses multiple IoT devices to simultaneously collect image information of the same object of interest, perform semantic feature extraction and compression encoding processing, and then use one or more edge servers to achieve precise edge computing, which can effectively improve the efficiency and accuracy of the semantic edge reasoning system in performing recognition tasks.

[0042] (2) The intelligent edge computing method for cross-scenario deployment proposed in this application mainly includes a two-step adaptation process, which uses an unsupervised domain adaptation strategy and a knowledge extraction strategy respectively. By utilizing the data correlation between the unlabeled data in the source domain and the target domain, it can adaptively handle changes in data distribution and wireless channel signal-to-noise ratio without relying on labeled data in the target domain, and can achieve rapid and economical large-scale deployment in dynamic application scenarios.

[0043] (3) The intelligent edge computing method for cross-scenario deployment proposed in this application, in the unsupervised domain adaptation stage, designs a total domain adaptation loss function that considers both the classification loss of source domain data and the domain adaptation loss of the target domain to the source domain; in the knowledge extraction stage, designs a total knowledge extraction loss function that considers both the classification loss of source domain data and the domain adaptation loss of the target domain to the source domain, as well as the cross entropy loss between the prediction results using the student model and the prediction results using the teacher model. These two loss functions fully consider the factors that cause losses in the two-step adaptation process, and design a loss function that can comprehensively and accurately reflect the corresponding loss of each step of the adaptation process, so that the student model obtained by training and fine-tuning is more accurate, so that the system can accurately perform prediction tasks in cross-scenario situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 This is a flow chart of the intelligent edge computing method for cross-scenario deployment of the present invention;

[0046] Figure 2 A schematic diagram of scenario 1 for deploying the semantic edge reasoning system of the present invention;

[0047] Figure 3 A schematic diagram of scenario 2 for deployment of the semantic edge reasoning system of the present invention;

[0048] Figure 4 It is a schematic diagram of the structure module of the semantic edge reasoning system of the present invention;

[0049] Figure 5 Some images in the Amazon image dataset;

[0050] Figure 6 It is a schematic diagram of the processing effect of the present invention on some images in the Amazon image dataset;

[0051] Figure 7 A comparison chart of the inference accuracy of the system of the present invention in different data distribution migration tasks;

[0052] Figure 8 It is a schematic diagram of the effect of the signal-to-noise ratio on the accuracy of the migration task when the compression rate is 0.1 according to the present invention;

[0053] Fig. 9Schematic diagram of the effect of the signal-to-noise ratio on the accuracy of the migration task when the compression rate is 0.25 according to the present invention;

[0054] Fig.10 This is a schematic diagram of the intelligent edge computing system module for cross-scenario deployment of the present invention;

[0055] Fig.11 It is a schematic diagram of the terminal structure of the present invention. DETAILED DESCRIPTION

[0056] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0057] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0058] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0059] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0060] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0061] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0063] The existing intelligent IoT system deployed with deep neural networks can use semantic communication technology to extract semantic features from sensor data locally through a semantic encoder, and then transmit the extracted semantic features to the edge server. After receiving the semantic features, the edge server uses them for task reasoning.

[0064] However, semantic communication based on deep learning is easily affected by dynamic environments. When the environment in which the wireless IoT device is located changes, the distribution of the sensor data collected by the wireless IoT device and the signal-to-noise ratio of the wireless channel will change. However, deep neural networks cannot use domain invariance to perform task reasoning in different deployment scenarios. In order to maintain satisfactory reasoning accuracy, independent deep neural network models are mainly trained offline at different signal-to-noise ratios. In actual applications, models pre-trained at the most realistic signal-to-noise ratio are used to cope with changes in the signal-to-noise ratio of the wireless channel. In addition, the model is trained by collecting enough labeled data samples in different deployment scenarios to cope with changes in data distribution. It can be seen that the existing technology cannot well guarantee the reasoning accuracy of the system when the distribution of sensor data and the signal-to-noise ratio of the wireless channel change, and cannot be deployed on a large scale when the number of samples is small.

[0065] Considering that transfer learning has the characteristic of repeatedly using deep neural network models to complete related tasks, it can avoid retraining the model and effectively reduce the online training time. Transfer learning includes an offline training phase in a familiar "source domain" and an online training phase in an unfamiliar "target domain". Few-sample learning is a common transfer learning method, but after deployment, few-sample learning requires collecting a small number of samples for fine-tuning.

[0066] Based on this, the present invention proposes an intelligent edge computing method for cross-scenario deployment, which mainly includes: the first step is to set the signal-to-noise ratio of the target domain and the source domain to be the same. At this time, only the impact of different data distribution caused by different deployment locations needs to be dealt with. Minimizing the local maximum average difference is used as part of the loss function to shorten the distance between the semantic features of the labeled data in the source domain and the semantic features of the unlabeled data in the target domain, so that the semantic edge reasoning system has "domain invariance"; the second step is to apply knowledge distillation, regard the deep neural network model trained in the first step as the teacher model, and design a student model with the same structure as the teacher model according to the actual wireless channel signal-to-noise ratio deployed in the new environment, and fine-tune the student model so that the semantic edge reasoning system can adapt to the changes in the wireless channel signal-to-noise ratio in the deployment environment. Through unsupervised domain adaptation and knowledge distillation, this method utilizes the correlation between labeled data in the source domain and unlabeled data in the target domain, and can achieve rapid and economical large-scale deployment of semantic edge reasoning systems in different application scenarios without the need for labeled samples in the target domain.

[0067] The embodiment of the present invention provides an intelligent edge computing method for cross-scenario deployment, which is deployed on electronic devices such as computers and servers and is applied to scenarios where semantic edge reasoning is performed using an intelligent Internet of Things system deployed based on a deep neural network model. Specifically, Figure 1 As shown, the steps of the method in this embodiment include:

[0068] Step S100: using a plurality of IoT devices to simultaneously collect image frames of the same object in a source domain or a target domain at different viewing angles to obtain source domain data and target domain data;

[0069] Specifically, this embodiment divides different scenarios into a source domain and a target domain. A sufficient number of labeled true data can be collected in the source domain, while the target domain refers to the environment where the semantic edge reasoning system needs to be deployed. The data collected in this environment is basically unlabeled, that is, there is almost no data truth.

[0070] For example, Figure 2 and Figure 3The semantic edge reasoning system deployment scenarios 1 and 2 are shown in the figure. The semantic edge reasoning system consists of multiple Internet of Things (IoT) devices and an edge server. Multiple IoT devices can simultaneously capture video streams or image data of roadside objects (such as vehicles, pedestrians, etc.), extract semantic features from the collected data, and send them to the edge server for object recognition. The IoT devices in these two traffic scenarios are monitoring devices. Scenario 1 describes the scene on a rural road, where the surrounding environment is relatively open, there are fewer vehicles, and the distance between the IoT device and the edge server is relatively far; Scenario 2 describes the scene at an intersection, where the surrounding environment is relatively complex, there are more vehicles and pedestrians, and the distance between the IoT device and the edge server is relatively close. It can be seen from these two scenarios that when the deployment environment changes, the image data collected by the IoT device will be very different, that is, the data distribution will change; at the same time, since the distance between the IoT device and the edge server will also change, this will also cause the wireless channel signal-to-noise ratio to change.

[0071] The semantic edge reasoning system in this embodiment is constructed based on several IoT devices and at least one edge server. Several IoT devices are used to simultaneously collect video streams or a series of image frames of the same object of interest in different perspectives in the source domain or target domain, obtain source domain data and target domain data, and send the collected video streams or image frames to the edge server. Among them, the data in the source domain is the true value of the labeled data, and the data in the target domain is the unlabeled data; the edge server can be a cloud server, a high-performance computer server, etc., and the type of IoT device can be flexibly adjusted as the application field and application scenario change. The application fields include IoT, industrial automation, intelligent transportation, retail, medical health and urban management, etc. The present invention does not specifically limit the types of IoT devices and edge servers used to deploy the semantic edge reasoning system.

[0072] Step S200: Based on the source domain data and the target domain data, the semantic edge reasoning system is fine-tuned using an unsupervised domain adaptation algorithm and a knowledge extraction algorithm to obtain an optimized semantic edge reasoning system;

[0073] In one implementation, step S200 includes the following steps:

[0074] Step S210: aligning the source domain data and the target domain data using an unsupervised domain adaptation algorithm to obtain updated parameters of the semantic edge reasoning system;

[0075] Specifically, assuming that the signal-to-noise ratio is stable, the unsupervised domain adaptation algorithm is used to perform data distribution alignment operations on the source domain data and the target domain data in the semantic edge reasoning system, so that the semantic edge reasoning system can adapt to the difference in data distribution between the source domain and the target domain, thereby obtaining the updated parameters of the deep neural network model corresponding to the semantic edge reasoning system to improve the generalization ability of the system.

[0076] Step S220: Based on the updated parameters, the semantic edge reasoning system is fine-tuned according to the channel conditions of the target domain using a knowledge extraction algorithm to obtain an optimized semantic edge reasoning system.

[0077] Specifically, the knowledge extraction algorithm is used to fine-tune the semantic edge reasoning system. That is, the deep neural network model corresponding to the updated parameters is used as the teacher model, a student model with the same structure as the teacher model is constructed, and the student model is used to imitate the teacher model. After several rounds of fine-tuning of the semantic edge reasoning system, the optimized semantic edge reasoning system can ensure good reasoning accuracy under the channel conditions of the target domain.

[0078] This embodiment mainly uses an unsupervised domain adaptation algorithm and a knowledge extraction algorithm in two steps to adapt the semantic edge reasoning system to new data distribution and wireless communication channels.

[0079] Step S300: Use the optimized semantic edge reasoning system to perform edge computing on the target domain data to obtain a prediction result.

[0080] Specifically, the semantic understanding and edge computing capabilities of the optimized semantic edge reasoning system are used to perform semantic feature extraction, compression, and encoding and decoding processing on the target domain data to extract key information, and then use the key information to predict the recognition results of the object, including the object's type, posture, identification status and other information.

[0081] In this embodiment, the unsupervised domain adaptation algorithm is used to realize the ability of the semantic edge reasoning system to adapt to the data distribution of the target domain when there is no labeled data in the target domain, which is beneficial to improving the generalization ability of the system; the teacher model is applied to the student model and fine-tuned using the knowledge extraction algorithm so that it can accurately infer in the target domain. It can be seen that the semantic edge reasoning system optimized by the unsupervised domain adaptation algorithm and the knowledge extraction algorithm of the present invention can adapt well to changes in channel states and changes in the distribution of sensor data across scenarios, ensure the edge computing accuracy of the system, avoid dependence on labeled data, simplify the deployment process of the system, and can achieve large-scale deployment with a small number of samples, which can significantly improve the adaptability of the semantic edge reasoning system in different environments.

[0082] In a preferred embodiment, the step S210 uses an unsupervised domain adaptation algorithm to distribute and align the source domain data with the target domain data to obtain updated parameters corresponding to the semantic edge reasoning system, including:

[0083] Step S211: extracting semantic features and performing compression coding on the source domain data and the target domain data to obtain a source domain semantic coding sequence and a target semantic coding sequence;

[0084] Step S212: calculating the domain adaptation loss between the source domain semantic coding sequence and the target semantic coding sequence using an unsupervised domain adaptation algorithm;

[0085] Step S213: Based on the domain adaptation loss, the source domain data and the target domain data are distributed aligned to obtain updated parameters corresponding to the semantic edge reasoning system.

[0086] Specifically, Figure 4 Shown by The edge computing system consists of an IoT device and an edge server. The IoT devices are used to collaboratively monitor the target of interest from different viewpoints and send the observed image information to the edge server to derive inference results. The IoT devices are designed to collaboratively extract semantic features related to the reasoning task from the captured images. IoT devices are marked as , and the IoT devices obtain image representation through object detection ,in , and Respectively represent the number of channels, height and width of the image sample. Representing images The corresponding annotation, set Contains the categories of objects that are recognized to appear in the scene, such as "helmet", "bicycle", etc., representing the object depicted in the image. Parameterized Semantic feature extractor, captured image After semantic feature extractor It will generate semantic features ,in To further compress the data, we will use Parameterized Compression and Channel Coders Will Processed into compressed semantic features ,in The IoT device will then send the compressed semantic features to the edge server in an orthogonal wireless channel. , so concurrent transmissions will not interfere with each other. Normalize and consider the wireless channel with additive Gaussian white noise between the IoT device and the edge server. The channel output symbol received at the receiver is expressed as ,in , , represents the identity matrix, represents the normalized receiver noise power, The value of The channel gain between the device and the edge server is related to the Corresponding to the weaker channel. It is easy to understand that in different scenarios, the number of deployed IoT devices and edge servers can be flexibly adjusted, and ensure that the data collected by each IoT device can be received by at least one edge server, and each edge server can communicate with each other to ensure that all data in the source domain and the target domain can be processed by the edge server to ensure the comprehensiveness of data processing.

[0087] The edge server is used to simultaneously receive compressed semantic features collected from multiple IoT devices for the same object at different perspectives. ,in The edge server first sends multiple Merge to get the merged semantic features ,Then is entered into Parameterized Decoder , and get the prediction results , specifically:

[0088] ;

[0089] Among them, the prediction results The structure is , Indicates the total number of categories included in the prediction results. Finally, the edge server selects The maximum value in The corresponding index as image Reasoning Tags Therefore, the present invention creates a The zero vector of the label The corresponding position is set to 1, and the other positions are set to 0 to obtain a unique hot vector , and use the hot unique vector to represent the distribution of object categories corresponding to the prediction results.

[0090] The semantic edge reasoning system in this embodiment is mainly composed of three parts: semantic feature extractor, compression and channel encoder and decoder, which are expressed as .in, Represents the semantic feature extractor parameters. Since the deep residual network (ResNet) has a powerful image feature extraction capability, a deep residual network (such as ResNet-50) is selected as ,It needs to be stated that the structure and number of convolutional layers in the ,deep residual network can be adjusted accordingly according to the number of object ,classifications corresponding to the actual target domain scene, or some or all of the ,full convolutional layers can be replaced with recurrent neural network (RNN) layers, etc., to ensure that the ,semantic features of all types of objects in the target domain scene can be ,accurately extracted. represents the compression and channel encoder parameters, and in a preferred example uses a fully connected layer as , and define the compression ratio as .in, and Separate table Show The input and output dimensions of the data. On the basis of ensuring the accuracy of the overall reasoning, the compression rate can be adjusted to balance the retention of key information and the removal of redundant information. Denotes decoder parameters. In a preferred example, two linear layers and a Rectified Linear Unit (ReLU) layer are used as a decoder and output an inference result.

[0091] Based on the above Figure 4 The semantic edge reasoning system shown in step 1 in this embodiment uses an unsupervised domain adaptation algorithm to distribute and align the source domain data and the target domain data to obtain updated parameters corresponding to the semantic edge reasoning system, specifically including:

[0092] First, assuming that the signal-to-noise ratio of the wireless channel in the source domain and the target domain is the same, a semantic feature extractor is used to extract semantic features from the source domain data and the target domain data respectively, and the extracted semantic features are compressed and encoded using a compression and channel encoder to obtain the source domain semantic coding sequence and the target semantic coding sequence, that is, the source domain data and target domain data Input to The corresponding target domain sensing device and In the source domain, the semantic encoding sequence is generated and the target semantic encoding sequence ,in Considering that there is noise in the real wireless channel, the edge server combination will carry noise after passing through the wireless channel. Therefore, in this embodiment, the source domain semantic coding sequence and the target semantic coding sequence with added noise are respectively expressed as and .

[0093] Then, an unsupervised domain adaptation algorithm is used to calculate the domain adaptation loss between the source domain semantic coding sequence and the target semantic coding sequence, that is, to calculate the local maximum average difference between the source domain semantic coding sequence and the target semantic coding sequence; based on the domain adaptation loss, the source domain data and the target domain data are distributed aligned to obtain the updated parameters corresponding to the semantic edge reasoning system.

[0094] For example, the local maximum average difference is used to calculate the source domain semantic coding sequence and the target semantic coding sequence and The weighted difference between the categories is calculated as:

[0095]

[0096] in, and They represent the distributions that the source domain data and the target domain data obey respectively, and the distributions that the two obey do not affect each other; Indicates the prediction result Types of categories; The source domain data individual data; The target domain individual data; and Represent the number of source domain data and the number of target domain data respectively; and Represent the source domain dataset and the target domain dataset respectively; Indicates Source domain data represents categories c The weight of ; The source domain data One-hot vector of data No. elements; Indicates Source domain data represents categories c The weight of ; The source domain data One-hot vector of data No. elements; Represents the source domain dataset Inside data and true value; Indicates The target domain data represents the category c The weight of ; The target domain data The one-hot vector predicted by the decoder No. elements; The target domain data The one-hot vector predicted by the decoder No. elements; Indicates The target domain data represents the category c The weight of ; The target domain data The one-hot vector predicted by the decoder No. elements; The target domain data The one-hot vector predicted by the decoder No. elements; Indicates the source domain received by the edge server Data passed and The noisy semantic features after ; Indicates the source domain received by the edge server Data passed and The noisy semantic features after ; Indicates the target domain received by the edge server Data passed and The noisy semantic features after ; Indicates the target domain received by the edge server Data passed and The noisy semantic features after ; So is the reproducing kernel Hilbert space of the characteristic kernel, the function It is used to map data samples to the reproducing kernel Hilbert space. Finally, the domain adaptation loss is calculated based on the category weighted difference to align the data distribution of the source domain and the target domain by minimizing the domain adaptation loss.

[0097] Among them, based on the domain adaptation loss and the deep neural network model corresponding to the semantic edge reasoning system, the source domain data and the target domain data are distributed aligned to obtain the updated parameters corresponding to the semantic edge reasoning system, that is, the specific process of minimizing the domain adaptation loss to align the data distribution of the source domain and the target domain includes: decoding the source domain semantic coding sequence to obtain the prediction result of the source domain data; calculating the source domain cross entropy loss based on the prediction result of the source domain data and the corresponding true value; obtaining the total domain adaptation loss based on the source domain cross entropy loss and the domain adaptation loss; based on the total domain adaptation loss and the deep neural network model corresponding to the semantic edge reasoning system, the source domain data and the target domain data are distributed aligned to obtain the updated parameters of the deep neural network model corresponding to the semantic edge reasoning system.

[0098] For example, when using source domain data to calculate classification loss, the loss function can be constructed by formula (1) to obtain the prediction result of the source domain data and the true value of the source domain data sample:

[0099] ;

[0100] in, This embodiment calculates the source domain cross entropy loss according to the output result of the neural network and the corresponding true value, and adjusts the parameters of the deep neural network model corresponding to the semantic edge reasoning system according to the size of the source domain cross entropy loss.

[0101] Then, using the source domain data and target domain data Compute domain adaptation loss , the expression of domain adaptation loss is:

[0102] ;

[0103] in, and Respectively represent the distributions that the source domain data and the target domain data obey, and the distributions that the two obey do not affect each other. In this embodiment, the domain adaptation loss function is used to calculate the weighted difference between the compressed semantic features and the categories.

[0104] Finally, the total domain adaptation loss is expressed based on the source domain cross entropy loss and the domain adaptation loss. The total domain adaptation loss function in the unsupervised domain adaptation stage can be expressed as:

[0105] ;

[0106] in, Represents the weight. Since the total domain adaptation loss takes into account both the classification loss of the teacher model and the domain adaptation loss of the target domain to the source domain, when the total domain adaptation loss tends to be stable or reaches a preset iteration round, the source domain data and the target domain data distribution can be aligned to obtain the updated parameters of the deep neural network model corresponding to the semantic edge reasoning system, that is, to obtain the trained deep neural network model corresponding to the optimized semantic edge reasoning system.

[0107] During the training process in step 1, the source domain dataset needs to be input ; Target domain dataset , the batch size of training is set to , the total number of training rounds is set to , initialize network parameters At the beginning of the round, sampling , and then calculate the loss function through formula (5) , according to the calculated loss function Update the model: , that is, the process of adjusting the system's deep neural network model parameters according to the gradient information during iterative training with the goal of minimizing the loss function. is the learning rate, completing the first round of training. Repeat the steps of the first round of training until the training round ends, and output the final trained deep neural network model.

[0108] In a preferred embodiment, the step S220 uses a knowledge extraction algorithm to fine-tune the semantic edge reasoning system according to the channel conditions of the target domain based on the updated parameters to obtain an optimized semantic edge reasoning system, including:

[0109] Step S221: constructing a teacher model and a student model with the same structure based on the deep neural network model corresponding to the updated parameters;

[0110] Step S222: Based on the deep neural network model corresponding to the updated parameters, the source domain data and the target domain data, the student model is fine-tuned according to the channel conditions of the target domain using a knowledge extraction algorithm to obtain an optimized student model;

[0111] Step S223: Based on the optimized student model, an optimized semantic edge reasoning system is obtained.

[0112] Specifically, after aligning the source domain data and the target domain data distribution, this embodiment uses the knowledge distillation algorithm to solve the difference in channel conditions. The basic idea of ​​the knowledge distillation algorithm is to make the student model imitate the teacher model, and after several rounds of fine-tuning, the system maintains good reasoning accuracy under the changing target domain channel conditions. Figure 4 As shown in step 2 in , firstly, the deep neural network model corresponding to the updated parameters is used as the teacher model, and a student model with the same structure as the teacher model is constructed, that is, the parameters are The teacher model and parameters are The student model and initialization settings .in, It represents the deep neural network model corresponding to the target domain after the training converges in step 1. The teacher model refers to the deep neural network model corresponding to the semantic edge reasoning system under the condition that the channel conditions in the source domain and the target domain are the same and the feature distributions of the source domain data and the target domain data are aligned; the student model refers to the deep neural network model corresponding to the semantic edge reasoning system under the condition that the channel conditions in the source domain and the target domain are the same and the feature distributions of the source domain data and the target domain data are aligned, and the channel environment in the target domain changes dynamically.

[0113] Then, based on the teacher model and the aligned target domain data, the knowledge extraction algorithm is used to fine-tune the student model according to the channel conditions of the target domain to obtain the optimized student model;

[0114] Specifically, the training process of the teacher model belongs to the prior art and will not be repeated here. Based on the teacher model, the first prediction result of the target domain data is calculated, that is, the target domain data is predicted using the teacher model; the target domain data is screened based on the confidence of the first prediction result to retain the target domain data samples that are valuable for reliable predictions and delete the target domain data corresponding to unreliable predictions, thereby obtaining the screened target domain data; based on the student model, the second prediction result of the screened target domain data is calculated, that is, the target domain data is predicted using the student model; based on the first prediction result and the second prediction result, the cross-entropy loss between the prediction results of the target domain data by the teacher model and the student model is calculated, and the student model is fine-tuned according to the channel conditions of the target domain and the cross-entropy loss value to obtain an optimized student model, so that the optimized student model can accurately perform the prediction task under the channel conditions of the target domain. Among them, the process of fine-tuning based on the first prediction result and the second prediction result specifically includes: calculating the cross entropy of the first prediction result and the second prediction result to obtain the knowledge extraction loss; based on the source cross entropy loss, the domain adaptation loss and the knowledge distillation loss, the total knowledge extraction loss is obtained; based on the total knowledge extraction loss, the student model is fine-tuned according to the channel conditions of the target domain to obtain an optimized student model.

[0115] In one exemplary embodiment, taking into account changing channel conditions The labels of the data samples will not be changed, so this embodiment allows the teacher model Get the target domain dataset The prediction results To guide students model . The reliability of The incorrect output of the student model Therefore, in order to enhance the robustness of knowledge distillation, this embodiment uses confidence as the evaluation criterion for prediction results and uses a binary mask to filter unreliable predictions. Based on formula (1), Corresponding to the teacher model Generated The maximum value in . The higher the value, the more accurate the prediction result. The higher the probability. Prediction results input to the teacher model Exceeding confidence threshold , the result is considered reliable and set Mask On the contrary, when Below confidence threshold , the result is considered unreliable and The mask is set to , so as to exclude the target domain data sample in this round of fine-tuning. It is easy to understand that the source domain data and the target domain data in the present invention refer to the source domain data set and the target domain data set, that is, they include multiple source domain data samples and target domain data samples respectively.

[0116] Then, after multiple rounds of fine-tuning, only the target domain data samples with mask equal to 1 are retained, and the prediction results of the teacher model and the student model on the target domain data are expressed as and By calculating and The cross entropy between them gives the knowledge distillation loss, namely:

[0117] ;

[0118] The total knowledge extraction loss during fine-tuning using the knowledge distillation algorithm is used to optimize the student model to obtain an optimized student model, where the expression of the total knowledge extraction loss is:

[0119] ;

[0120] in, represents the cross entropy loss of source domain data, represents the unsupervised domain adaptation loss, represents the knowledge distillation loss, and Since the total knowledge extraction loss takes into account the classification loss of the teacher model and the domain adaptation loss of the target domain to the source domain, as well as the cross entropy loss between the prediction results using the student model and the prediction results using the teacher model, when the total knowledge extraction loss reaches the preset total knowledge extraction loss threshold, a student model with good performance can be obtained.

[0121] During the training process in step 2, the source domain dataset needs to be input ; Target domain dataset , the batch size of training is set to , the total number of training rounds is set to , initialize network parameters At the beginning of the round, sampling , and then calculate the loss function through formula (7) ; According to the calculated loss function Student Update Model: , that is, the process of adjusting the student model parameters according to the gradient information during iterative training with the goal of minimizing the loss function. is the learning rate, and the first round of training is completed. Repeat the steps of the first round of training until the training After the round ends, the final trained student model is output.

[0122] Combining steps 1 and 2, we can see that these two steps use unsupervised domain adaptation and knowledge extraction techniques respectively to adapt the semantic edge reasoning system to new data distributions and channels. Step 1 uses unsupervised domain adaptation technology, without the need for labeled data in the target domain, so that the deep neural network in the source domain can adapt to changes in data distribution between the source domain and the target domain; Step 2 uses the model trained in step 1 as the teacher model, and further fine-tunes the student model in the target domain in combination with knowledge distillation to adapt to the actual signal-to-noise ratio.

[0123] In order to verify the effectiveness of the method proposed in the present invention and the accuracy of edge computing, the following experimental simulation is performed, specifically:

[0124] The simulation experiment is conducted using the OFFICE-31 image dataset. This dataset includes images of 31 common office supplies, and each object has three different image styles: the Amazon image dataset (A) has 2817 images, the low-resolution image dataset (W) captured by a webcam has 795 images, and the high-resolution image dataset (D) captured by a DSLR camera has 498 images. Figure 5The following are some of the image data from the Amazon image dataset. The W and D datasets depict the same office objects but at different resolutions. The images in A have different item styles and backgrounds than the images in W and D. This simulation experiment uses these three sets of image datasets to simulate the changes in data distribution in different deployment scenarios. For example, the transfer task from source domain A to target domain W is represented as A→W. All images are cropped to 224×224 three-channel pixels. In order to simulate multi-view images taken by different IoT cameras, Figure 6 The image shown is divided into four parts and overlapped at 150×150 pixels.

[0125] In order to show the effectiveness of the designed unsupervised domain adaptation in dealing with changes in data distribution, this experiment first assumes that the target and source domains have the same channel conditions and the signal-to-noise ratio of all links is 5dB. Then only step 1 of the present invention needs to be used. The experimental results are shown in Figure 7 As shown. It can be seen that when different methods are used, the reasoning accuracy of 6 different cross-domain tasks is different; W and D have similar data distribution but different resolutions, and the unsupervised domain adaptation performance exceeds 95% for both W→D and D→W tasks, and even achieves 99.78% accuracy in the W→D task. For tasks A→W and A→D, the unsupervised domain adaptation proposed by the present invention achieves 85.11% and 84.43% accuracy respectively; for tasks D→A and W→A, the unsupervised domain adaptation proposed by the present invention achieves 63.4% and 62.15% accuracy, because D or W has much fewer images than A.

[0126] Then, the experimental performance of step 1 and step 2 is comprehensively evaluated to solve the changes in data distribution and wireless channel signal-to-noise ratio. For the change in data distribution, this experiment uses the A→W task for analysis. First, the A→W task step 1 is trained under the condition that the signal-to-noise ratio is equal to 5dB. Then, the model trained in step 1 is used as the teacher model to train step 2 in a new environment to adapt to the changes in the signal-to-noise ratio of the wireless channel in the new environment. Step 1+step 2 are all the steps of the method of the present invention. The signal-to-noise ratio of the training process in step 1 is 5dB. Step 2 uses the model trained in step 1 as the teacher model to help the student model in an environment where the signal-to-noise ratio changes to adapt to the changes in the signal-to-noise ratio. Figure 8 and Fig. 9 The experiment corresponding to step 1+step 2 in the above example refers to testing the accuracy of the target domain data in an environment with a varying signal-to-noise ratio using the student model trained in step 2. Figure 8 and Fig. 9 The experiment corresponding to step 1 in the above example is to use the model trained in step 1 to test the accuracy of the target domain data in a new environment; Figure 8 and Fig. 9The direct test experiment in this paper refers to training the model with only source domain data when the signal-to-noise ratio is 5dB. Specifically, it refers to testing the accuracy of the target domain data in a new environment using the newly trained model with source domain data. The test results are as follows: Figure 8 and Fig. 9 shown.

[0127] Figure 8 The figure shows the inference accuracy under different signal-to-noise ratios when the compression rate is set to 0.1. It can be seen that the inference accuracy decreases under lower signal-to-noise ratios. Nevertheless, compared with the model directly trained in step 1 and tested directly in a new environment without any fine-tuning, the present invention has a good performance improvement, especially when the SNR of the target domain is lower than -10dB. Fig. 9 The figure shows the inference accuracy at different signal-to-noise ratios when the compression rate is set to 0.25. Figure 8 In terms of Fig. 9 The experimental performance is better under low signal-to-noise ratio, which means that when the signal-to-noise ratio is poor, the performance of the system can be improved by increasing the compression rate.

[0128] like Fig.10 As shown, corresponding to the above-mentioned intelligent edge computing method for cross-scenario deployment, an embodiment of the present invention further provides an intelligent edge computing system for cross-scenario deployment, and the above-mentioned intelligent edge computing system for cross-scenario deployment includes:

[0129] The data acquisition module 101 is used to use a number of IoT devices to simultaneously acquire image frames of the same object in a source domain or a target domain at different viewing angles to obtain source domain data and target domain data;

[0130] A system optimization module 102, configured to fine-tune the semantic edge reasoning system based on the source domain data and the target domain data using an unsupervised domain adaptation algorithm and a knowledge extraction algorithm to obtain an optimized semantic edge reasoning system, wherein the semantic edge reasoning system is constructed based on the IoT device and at least one edge server;

[0131] The prediction module 103 is used to use the optimized semantic edge reasoning system to perform edge computing on the target domain data to obtain a prediction result.

[0132] Specifically, in this embodiment, the specific functions of the above-mentioned intelligent edge computing system for cross-scenario deployment can also refer to the corresponding description in the above-mentioned intelligent edge computing method for cross-scenario deployment, which will not be repeated here.

[0133] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be as follows: Fig.11As shown. The terminal can be used to execute the intelligent edge computing method for cross-scenario deployment provided in the above embodiment. For the sake of brevity, it is not repeated here. The terminal includes: a processor, the processor is coupled to a memory, the memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions stored in the memory, so that the method in the above method embodiment is executed.

[0134] The present invention also provides a computer-readable storage medium on which computer instructions for implementing the method in the above method embodiment are stored.

[0135] For example, when the computer program is executed by a computer, the computer can implement the method in the above method embodiment.

[0136] The embodiment of the present application also provides a computer program product including instructions, which, when executed by a computer, enables the computer to implement the method in the above method embodiment.

[0137] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0142] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

Claims

1. An intelligent edge computing method for cross-scenario deployment, characterized in that: The scenario of semantic edge reasoning applied to intelligent IoT systems deployed based on deep neural network models mainly includes the following steps: Use several IoT devices to simultaneously collect image frames of the same object in the source domain or target domain at different perspectives to obtain source domain data and target domain data; Based on the source domain data and the target domain data, fine-tuning the semantic edge reasoning system using an unsupervised domain adaptation algorithm and a knowledge extraction algorithm to obtain an optimized semantic edge reasoning system, wherein the semantic edge reasoning system is constructed based on the Internet of Things device and at least one edge server; Using the optimized semantic edge reasoning system to perform edge computing on the target domain data to obtain a prediction result; The method of fine-tuning the semantic edge reasoning system based on the source domain data and the target domain data using an unsupervised domain adaptation algorithm and a knowledge extraction algorithm to obtain an optimized semantic edge reasoning system includes: Extracting semantic features and compressing the source domain data and the target domain data to obtain a source domain semantic coding sequence and a target semantic coding sequence; calculating the category weighted difference between the source domain semantic coding sequence and the target semantic coding sequence using the local maximum average difference as the domain adaptation loss; Decoding the source domain semantic coding sequence to obtain a prediction result of the source domain data; calculating a source domain cross entropy loss based on the prediction result of the source domain data and a corresponding true value; obtaining a total domain adaptation loss based on the source domain cross entropy loss and the domain adaptation loss; and aligning the distribution of the source domain data and the target domain data based on the total domain adaptation loss to obtain updated parameters corresponding to the semantic edge reasoning system; The deep neural network model corresponding to the updated parameters is used as a teacher model to construct a student model with the same structure as the teacher model; based on the deep neural network model corresponding to the updated parameters, the source domain data and the target domain data, the student model is fine-tuned according to the channel conditions of the target domain using a knowledge extraction algorithm to obtain an optimized student model; based on the optimized student model, an optimized semantic edge reasoning system is obtained; wherein the teacher model refers to the deep neural network model corresponding to the semantic edge reasoning system under the condition that the channel conditions in the source domain and the target domain are the same and the feature distributions of the source domain data and the target domain data are aligned; the student model refers to the deep neural network model corresponding to the semantic edge reasoning system under the condition that the channel conditions in the source domain and the target domain are the same and the feature distributions of the source domain data and the target domain data are aligned, and the channel environment of the target domain changes dynamically; The method of fine-tuning the student model according to the channel condition of the target domain by using the knowledge extraction algorithm to obtain an optimized student model includes: The student model is optimized using the total knowledge extraction loss to obtain an optimized student model, wherein the total knowledge extraction loss is obtained based on the source domain cross entropy loss, the domain adaptation loss and the knowledge distillation loss, and the knowledge distillation loss is obtained based on the target domain data samples corresponding to the confidence threshold of the prediction results of the teacher model for the target domain data set, and the cross entropy between the prediction results of the teacher model and the student model for the target domain data samples is calculated.

2. An intelligent edge computing system for cross-scenario deployment, characterized in that: The system is applied to implement the steps of the intelligent edge computing method for cross-scenario deployment as claimed in claim 1, and includes: A data acquisition module, used to use a number of IoT devices to simultaneously acquire image frames of the same object in a source domain or a target domain at different viewing angles to obtain source domain data and target domain data; A system optimization module, configured to fine-tune the semantic edge reasoning system based on the source domain data and the target domain data using an unsupervised domain adaptation algorithm and a knowledge extraction algorithm to obtain an optimized semantic edge reasoning system, wherein the semantic edge reasoning system is constructed based on the Internet of Things device and at least one edge server; A prediction module is used to perform edge computing on the target domain data using the optimized semantic edge reasoning system to obtain a prediction result.

3. A terminal, characterized in that: include: A memory for storing executable instructions; A processor, used to call and run the executable instructions in the memory to execute the steps of the intelligent edge computing method for cross-scenario deployment as described in claim 1.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions. When the program instructions are executed by the processor, the intelligent edge computing method for cross-scenario deployment as described in claim 1 is implemented.

Images