Image prediction method and device based on edge cloud architecture, network equipment and medium

Abstract

The present disclosure provides an image prediction method and device based on an edge cloud architecture, network equipment and a medium, relating to the technical field of edge cloud. The image prediction method based on the edge cloud architecture comprises: in response to the acquired to-be-predicted image, performing a feature extraction operation on the to-be-predicted image based on a feature extractor of an image prediction model to generate a feature data set; segmenting the feature data set into a first group of features and a second group of features based on an importance estimation operation; performing a local inference operation on the first group of features to generate a first inference result; sending the second group of features to a cloud in the edge cloud architecture for the cloud to perform a cloud inference operation on the second group of features to generate a second inference result; and receiving the second inference result sent by the cloud to fuse the first inference result and the second inference result to obtain a prediction result of the to-be-predicted image. Through the technical solution of the present disclosure, while ensuring the prediction accuracy, the local resource occupation and inference delay of the edge node can be reduced.

Description

Technical Field

[0001] This disclosure relates to the field of edge cloud technology, and in particular to an image prediction method based on an edge cloud architecture, an image prediction device based on an edge cloud architecture, a network device, and a computer-readable storage medium. Background Technology

[0002] In business scenarios such as smart cities or smart transportation, the deployment of applications extends from the cloud center to the edge. As the number of edge devices in the edge-cloud system continues to expand, the limitations of edge computing, storage, and network resources make it impossible for edge devices to complete high-precision model inference.

[0003] To address the aforementioned issues, feature segmentation operations were introduced. However, traditional neural network segmentation operations can significantly impact the performance of the original neural network, failing to fully meet the accuracy requirements of business scenarios such as smart cities.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide an image prediction method, apparatus, network device, and storage medium based on an edge cloud architecture, which at least to some extent overcomes the problem that edge devices cannot perform high-precision model inference in related technologies.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of this disclosure, an image prediction method based on an edge cloud architecture is provided, applied to edge nodes in the edge cloud architecture, comprising: responding to an acquired image to be predicted, performing a feature extraction operation on the image to be predicted using a feature extractor based on an image prediction model to generate a feature data set; dividing the feature data set into a first set of features and a second set of features based on an importance estimation operation; performing a local inference operation on the first set of features to generate a first inference result; sending the second set of features to the cloud in the edge cloud architecture so that the cloud performs a cloud inference operation on the second set of features to generate a second inference result; receiving the second inference result sent by the cloud to perform a fusion operation on the first inference result and the second inference result to obtain a prediction result for the image to be predicted.

[0008] In one embodiment, the segmentation of the feature data set into a first group of features and a second group of features based on the importance estimation operation includes: performing a classification and recognition operation on the feature data set based on the local classifier of the image prediction model to obtain the type information of the image to be predicted; performing an importance estimation operation based on the feature data set and the type information to obtain the importance of multiple feature data in the feature data set; sorting the multiple feature data from high to low importance; and segmenting the feature data set into a first group of features and a second group of features based on the sorting result.

[0009] In one embodiment, the step of performing an importance estimation operation based on the feature data set and the type information to obtain the importance of multiple feature data in the feature data set includes: performing spatial feature mapping on the feature data to obtain spatial feature information; inputting the spatial feature information and the type information into the interpretability model of the image prediction model to calculate the integral gradient of each feature data; and performing fitting analysis on the integral gradients of the multiple feature data to obtain the importance of the multiple feature data based on the fitting results.

[0010] In one embodiment, the step of performing fitting analysis on the integral gradients of the plurality of feature data to obtain the importance of the plurality of feature data based on the fitting results includes: performing fitting analysis on the plurality of integral gradients to obtain the fitting results, the fitting results being used to characterize the importance distribution; if it is detected that the importance distribution conforms to the target importance distribution, then the plurality of integral gradients are determined as the importance of the plurality of feature data; if it is detected that the importance distribution does not conform to the target importance distribution, then the deviation of the importance distribution relative to the target importance distribution is calculated based on the loss function of the image prediction model; the model parameters of the feature extractor and the local classifier are optimized based on the deviation, so as to update the integral gradient of each feature data based on the optimized feature extractor and the local classifier, and the updated integral gradient is determined as the importance of the plurality of feature data.

[0011] In one embodiment, performing local inference on the first set of features to generate a first inference result includes completing the first set of features based on a random mask model and the number of features in the feature data set to obtain a first set of inference features; and inputting the first set of inference feature data into the local classifier to generate the first inference result based on the classification prediction of the local classifier.

[0012] In one embodiment, before sending the second set of features to the cloud in the edge cloud architecture, the method further includes: compressing the second set of features and sending the compressed second set of features to the cloud.

[0013] In one embodiment, fusing the first inference result and the second inference result to obtain the prediction result of the image to be predicted includes: determining a first weight value based on the importance of the first set of features; determining a second weight value based on the importance of the second set of features; and adding and fusing the first inference result and the second inference result based on the first weight value and the second weight value to obtain the prediction result.

[0014] According to another aspect of this disclosure, an image prediction method based on an edge cloud architecture is provided, applied to the cloud in the edge cloud architecture, comprising: receiving a second set of features sent by an edge node in the edge cloud architecture; performing cloud inference operations on the second set of features to generate a second inference result; and sending the second inference result to the edge node.

[0015] In one embodiment, receiving the second set of features sent by the edge node in the edge cloud architecture includes: receiving compressed data sent by the edge node; and performing a decompression operation on the compressed data to obtain the second set of features.

[0016] In one embodiment, performing cloud-based inference operations on the second set of features to generate a second inference result includes: completing the second set of features based on a random mask model and the number of features in the feature data set to obtain a second set of inference features; and inputting the second set of inference feature data into a cloud classifier to generate the second inference result based on the classification prediction of the cloud classifier.

[0017] According to another aspect of this disclosure, an image prediction apparatus based on an edge cloud architecture is provided, applied to an edge node in the edge cloud architecture, comprising: a feature extraction module, configured to perform feature extraction operations on the image to be predicted based on an image prediction model feature extractor in response to an acquired image to be predicted, generating a feature data set; a segmentation module, configured to segment the feature data set into a first set of features and a second set of features based on an importance estimation operation; a first inference module, configured to perform local inference operations on the first set of features, generating a first inference result; a first sending module, configured to send the second set of features to the cloud in the edge cloud architecture, so that the cloud performs cloud inference operations on the second set of features, generating a second inference result; and a fusion module, configured to receive the second inference result sent by the cloud, and to perform a fusion operation on the first inference result and the second inference result to obtain a prediction result for the image to be predicted.

[0018] According to another aspect of this disclosure, an image prediction device based on an edge cloud architecture is provided, applied to the cloud in the edge cloud architecture, comprising: a receiving module for receiving a second set of features sent by an edge node in the edge cloud architecture; a second inference operation module for performing cloud inference operations on the second set of features to generate a second inference result; and a second sending module for sending the second inference result to the edge node.

[0019] According to another aspect of this disclosure, a network device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; the processor being configured to perform the image prediction method based on an edge cloud architecture described in the first aspect by executing the executable instructions.

[0020] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described image prediction method based on an edge cloud architecture.

[0021] The image prediction scheme based on an edge cloud architecture provided in the embodiments of this disclosure divides the feature data of the image to be predicted into a first set of features and a second set of features by estimating image features based on feature importance. A first inference result is obtained by performing local inference based on the first set of features, and a second inference result is obtained by performing cloud inference based on the second set of features, thereby realizing edge-cloud collaborative image prediction. The first and second inference results are further fused to obtain the prediction result of the image to be predicted. While ensuring prediction accuracy, on the one hand, completing part of the inference operation in the cloud can reduce the occupation of local resources. On the other hand, sending part of the data to the cloud for cloud inference can reduce cloud data transmission, reduce inference latency, and help ensure data security.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A schematic diagram of an edge cloud architecture according to an embodiment of this disclosure is shown;

[0025] Figure 2This diagram illustrates a flowchart of an image prediction method based on an edge cloud architecture according to an embodiment of the present disclosure.

[0026] Figure 3 This invention discloses a flowchart of another image prediction method based on an edge cloud architecture in an embodiment of the present invention.

[0027] Figure 4 This invention discloses a flowchart of another image prediction method based on an edge cloud architecture in an embodiment of the present disclosure.

[0028] Figure 5 This invention discloses a flowchart of another image prediction method based on an edge cloud architecture in an embodiment of the present disclosure.

[0029] Figure 6 This invention discloses a flowchart of another image prediction method based on an edge cloud architecture in an embodiment of the present disclosure.

[0030] Figure 7 This diagram illustrates a schematic block diagram of an image prediction system based on an edge cloud architecture according to an embodiment of the present disclosure.

[0031] Figure 8 This diagram illustrates a flowchart of an image prediction method based on an edge cloud architecture according to an embodiment of the present disclosure.

[0032] Figure 9 This diagram illustrates an image prediction device based on an edge cloud architecture according to an embodiment of the present disclosure.

[0033] Figure 10 This illustration shows a schematic diagram of another image prediction device based on an edge cloud architecture in an embodiment of this disclosure;

[0034] Figure 11 A structural block diagram of a computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0036] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] The image prediction scheme based on edge cloud architecture proposed in this disclosure mainly addresses the following problems existing in traditional edge-cloud collaboration:

[0038] Since edge-cloud collaboration architectures typically employ the method of dividing the neural network into multiple different parts and deploying them separately on edge-cloud devices to achieve edge-cloud collaboration, the direct division of the neural network can lead to a decrease in prediction accuracy.

[0039] In addition, if all data is transmitted to the cloud for processing, it can easily cause traffic congestion.

[0040] Figure 1 A schematic diagram of an edge cloud architecture according to an embodiment of this disclosure is shown.

[0041] like Figure 1 As shown, the edge cloud architecture includes a cloud platform 102, edge computing nodes 104, and application terminals 106.

[0042] Among them, cloud 102 is the central node of traditional cloud computing, cloud data center, edge computing node 104 is the edge side of cloud computing, which is divided into infrastructure edge and device edge, and application end 106 is terminal device, such as mobile phone, intelligent electrical equipment, various sensors, cameras, etc.

[0043] In one embodiment of this disclosure, the application terminal 106 is used to send the image to be predicted, the edge computing node 104 is used to perform local inference operations, and the cloud 102 performs cloud inference operations.

[0044] In another embodiment of this disclosure, the terminal device can perform local inference operations as an edge node.

[0045] The following will describe in more detail the steps of the image prediction method based on edge cloud architecture in this exemplary embodiment, with reference to the accompanying drawings and embodiments.

[0046] Figure 2 A flowchart of an image prediction method based on an edge cloud architecture is shown in an embodiment of this disclosure.

[0047] like Figure 2As shown, an image prediction method based on an edge cloud architecture according to an embodiment of this disclosure is applied to edge nodes in an edge cloud architecture, including:

[0048] In step S202, in response to the acquired image to be predicted, the feature extractor based on the image prediction model performs feature extraction operation on the image to be predicted to generate a feature data set.

[0049] Specifically, the image prediction model can be a convolutional neural network (CNN) model.

[0050] The image to be predicted is obtained, specifically by obtaining the pixel matrix of the image through the input layer of a convolutional neural network model.

[0051] The feature extractor of the image prediction model can be a convolutional layer in a convolutional neural network. Specifically, based on the convolution operation of the convolutional layer, the image features are extracted. Considering that the processing resources of edge nodes are limited and cannot perform fine-grained inference on the image, the image collected by the edge nodes is input into the convolutional layer of the neural network for preprocessing.

[0052] Step S204: Based on the importance estimation operation, the feature data set is divided into a first group of features and a second group of features.

[0053] Among them, by performing an importance estimation operation, the extracted

[0054] Step S206: Perform local inference operations on the first set of features to generate the first inference result.

[0055] Step S208: The second set of features is sent to the cloud in the edge cloud architecture so that the cloud can perform cloud inference operations on the second set of features and generate a second inference result.

[0056] Specifically, by estimating the importance of features, the book features extracted by the feature extractor of the image prediction model are ranked. Considering the business's requirements for credible inference of important features, the top K features are selected as the first group of features, and the remaining features are selected as the second group of features. The first group of features, which are important features, completes inference and prediction at the edge nodes, while the second group of features, which are non-important features, are transmitted to the cloud to complete inference and prediction.

[0057] Step S210: Receive the second inference result sent from the cloud, and perform a fusion operation on the first and second inference results to obtain the prediction result of the image to be predicted.

[0058] In this embodiment, by estimating image features based on feature importance, the feature data of the image to be predicted is divided into a first set of features and a second set of features. A first inference result is obtained by performing local inference based on the first set of features, and a second inference result is obtained by performing cloud inference based on the second set of features, thereby realizing edge-cloud collaborative image prediction. The first and second inference results are then fused to obtain the prediction result of the image to be predicted. While ensuring prediction accuracy, on the one hand, completing part of the inference operation in the cloud can reduce the occupation of local resources. On the other hand, sending part of the data to the cloud for cloud inference can reduce the transmission of cloud data, reduce inference latency, and help ensure data security.

[0059] like Figure 3 As shown, in one embodiment, step S104, a specific implementation of dividing the feature data set into a first group of features and a second group of features based on the importance estimation operation, includes:

[0060] Step S302: The local classifier based on the image prediction model performs classification and recognition operations on the feature data set to obtain the type information of the image to be predicted.

[0061] Specifically, the classifier can include pooling layers, activation function layers, fully connected layers, and softmax layers in a convolutional neural network. The pooling operation takes the overall statistical features of the neighboring regions of a certain position in the input matrix as the output of that position. After multiple rounds of processing by convolutional and pooling layers, the classification result is output by the fully connected layer at the end of the convolutional neural network. The probability distribution of the current image to be predicted belonging to different types can be obtained through the softmax layer.

[0062] Step S304: Perform importance estimation based on the feature data set and type information to obtain the importance of multiple feature data in the feature data set.

[0063] Importance can be calculated by integrating gradients, or by viewing the class activation map (CAM) to obtain the image heatmap used for final classification, which represents the importance of image features. Alternatively, tree models such as XGboost and RandomForest can output the importance of features after training.

[0064] Step S306: Sort the importance of multiple feature data from high to low.

[0065] Step S308: Based on the sorting results, the feature data set is divided into a first group of features and a second group of features.

[0066] One way to divide the feature data set into a first group and a second group is to set an importance threshold, with features greater than the threshold assigned to the first group and features less than or equal to the threshold assigned to the second group. Another way to divide the feature data set into a first group and a second group is to determine the segmentation scheme based on the prediction of the information gain after feature segmentation.

[0067] In this embodiment, the type information of the image to be predicted is obtained based on the classifier, and the importance of each feature data is estimated based on the type information. According to the obtained importance results, the features are sorted from high to low based on their importance. Based on the special importance estimation, the features are divided into a first group of features, namely important features, and a second group of features, namely unimportant features. The important features are inferred at the edge, which helps to ensure the security of the data, while the unimportant features are transmitted to the cloud for prediction.

[0068] In one embodiment, an importance estimation operation is performed based on a feature data set and type information to obtain the importance of multiple feature data in the feature data set, including:

[0069] Spatial feature information is obtained by mapping the feature data to spatial features.

[0070] Spatial feature information and type information are input into the interpretability model of the image prediction model to compute the integral gradient of each feature data.

[0071] In the field of machine learning, interpretability refers to explaining the principles of an algorithm or the basis for its results.

[0072] Spatial and type information are input into the interpretability model of the image prediction model. By calculating the integral gradient value, the importance of each spatial feature information to the classification type can be detected, thus obtaining the importance. Specifically, a completely black image (pixel intensity of 0) is used as the baseline input, and the category with the highest score in the output prediction category is used for attribution. The integral gradient algorithm is used to study which pixels, as feature data, provide the most important contribution when the network makes predictions.

[0073] The integral gradients of multiple feature data are fitted and analyzed to determine the importance of the multiple feature data based on the fitting results.

[0074] Among them, data fitting, also known as curve fitting, is a process that uses fitting operations to construct a continuous curve from the integral gradient of discrete feature data.

[0075] In this embodiment, the integral gradient of the feature data is calculated as the importance, and the integral gradient is fitted and analyzed to detect whether it meets the target requirements. This allows the integral gradient of the feature data to be obtained based on the detection and analysis results, ensuring the reliability of the obtained integral gradient.

[0076] In one embodiment, fitting analysis is performed on the integral gradients of multiple feature data to obtain the importance of the multiple feature data based on the fitting results. This includes: performing fitting analysis on multiple integral gradients to obtain fitting results, which are used to characterize the importance distribution; if the importance distribution is detected to conform to the target importance distribution, then the multiple integral gradients are determined as the importance of the multiple feature data; if the importance distribution is detected to not conform to the target importance distribution, then the deviation of the importance distribution from the target importance distribution is calculated based on the loss function of the image prediction model.

[0077] Among them, the loss function is a computational function used to measure the degree of difference between the model's predicted value f(x) and the true value Y. The smaller the loss function, the better the robustness of the model. The loss function based on distance metric usually maps the input data to a feature space based on distance metric, such as Euclidean space, Hamming space, etc. The mapped samples are regarded as points in the space. A suitable loss function is used to measure the distance between the true value of the sample and the model's predicted value in the feature space. Equation (1) shows one way of representing the loss function.

[0078] (1)

[0079] The smaller the distance between two points in the feature space, the better the model's predictive performance.

[0080] The model parameters of the bias-optimized feature extractor and local classifier are used to update the integral gradient of each feature data based on the optimized feature extractor and local classifier, and the updated integral gradient is used to determine the importance of multiple feature data.

[0081] In this embodiment, if the importance distribution is detected to be inconsistent with the target importance distribution, it indicates that there is a deviation in the currently calculated importance. The deviation of the importance distribution from the target importance distribution is calculated using the loss function based on the image prediction model, which affects the model parameters of the feature extractor and the local classifier. The importance of the feature data is then recalculated based on the optimized feature extractor and local classifier to ensure the accuracy of the importance.

[0082] like Figure 4 As shown, another embodiment of the image prediction method based on an edge cloud architecture according to this disclosure specifically includes:

[0083] Step S402: Select an image that needs to be detected for inference as the input to the feature extractor of the image prediction model.

[0084] Step S404: Process the input image using a feature extractor to obtain a feature data set.

[0085] Step S406: The feature data set obtains understandable image information, i.e., image type information, through the classifier of the remaining image prediction model.

[0086] In step S408, using features and image information as input, the integral gradient of the extracted feature mapping is calculated to obtain the importance, and the feature data in the feature data set is sorted according to the importance.

[0087] Step S410: Perform a fitting analysis on the feature importance. If the target requirements are met, output the feature importance ranking. If the target requirements are not met, calculate the loss function of the image prediction model.

[0088] Step S412: Calculate the difference between the current importance distribution and the target importance distribution using the loss function, and update the weight parameters of the feature extractor and the remaining neural network layers to re-estimate the importance of the features.

[0089] In one embodiment, performing local inference on a first set of features to generate a first inference result includes: completing the first set of features based on a random mask model and the number of features in the feature data set to obtain a first set of inference features; and inputting the first set of inference feature data into a local classifier to generate the first inference result based on the classification prediction of the local classifier.

[0090] In this embodiment, the random masking model is used to randomly mask image patches with missing second set of features in the image, and then predict the masked area using the first set of features to perform normal inference operations and ensure the reliability of the first inference result.

[0091] In one embodiment, before sending the second set of features to the cloud in the edge cloud architecture, the method further includes: compressing the second set of features and sending the compressed second set of features to the cloud.

[0092] In this embodiment, by compressing and encoding the less important second set of feature data at the edge, the compressed second set of features is sent to the cloud, which helps to reduce network traffic transmission and achieve reliable data transmission.

[0093] In one embodiment, fusing the first inference result and the second inference result to obtain a prediction result for the image to be predicted includes: determining a first weight value based on the importance of a first set of features; determining a second weight value based on the importance of a second set of features; and adding and fusing the first inference result and the second inference result based on the first weight value and the second weight value to obtain a prediction result.

[0094] In this embodiment, a first weight value and a second weight value are determined based on the importance of the first set of features and the importance of the second set of features, respectively, so as to assign corresponding weight values ​​to the first inference result and the second inference result, thereby ensuring the rationality of the prediction result.

[0095] like Figure 5 As shown, an image prediction method based on an edge cloud architecture according to another embodiment of this disclosure is applied to the cloud in an edge cloud architecture, including:

[0096] Step S 502: Receive the second set of features sent by the edge node in the edge cloud architecture.

[0097] Step S 504: Perform cloud-based inference operations on the second set of features to generate a second inference result.

[0098] Step S 506: Send the second inference result to the edge node.

[0099] In this embodiment, the cloud receives a second set of features and performs cloud inference operations based on the second set of features to obtain a second inference result, thereby realizing edge-cloud collaborative image prediction operations. The first and second inference results are then fused to obtain the prediction result for the image to be predicted. While ensuring prediction accuracy, on the one hand, completing part of the inference operations in the cloud can reduce the occupation of local resources. On the other hand, sending part of the data to the cloud for cloud inference can reduce the transmission of cloud data, reduce inference latency, and help ensure data security.

[0100] In one embodiment, receiving a second set of features sent by an edge node in an edge cloud architecture includes: receiving compressed data sent by the edge node; performing a decompression operation on the compressed data to obtain the second set of features.

[0101] In one embodiment, cloud-based inference operations are performed on the second set of features to generate a second inference result, including:

[0102] The second set of features is completed based on the random mask model and the number of features in the feature data set, resulting in the second set of inference features;

[0103] The second set of inference feature data is input into the cloud classifier to generate a second inference result based on the classification prediction of the cloud classifier.

[0104] In this embodiment, the random masking model is used to randomly mask image patches with missing first set of features in the image, and then predict the masked areas using the second set of features to perform normal inference operations, ensuring the reliability of the second inference result.

[0105] like Figure 6 As shown, a prediction result fusion method based on a random masking mechanism according to an embodiment of this disclosure specifically includes:

[0106] In step S602, the edge nodes perform importance estimation on the features of the feature data set to obtain the importance of the feature data in the set, and sort the importance from high to low.

[0107] Step S604: The sorted feature data is divided into a first group of features and a second group of features through logical operations.

[0108] Step S606: Generate a random mask model, send the second set of features to the cloud for calculation based on the random mask model to obtain the second inference result, and calculate the first set of features at the edge nodes to obtain the first inference result.

[0109] Step S608: Add the two inference results together to obtain the final prediction result.

[0110] like Figure 7 As shown, an image prediction system based on an edge cloud architecture according to an embodiment of the present disclosure includes a front-end inference module 702, a feature segmentation module 704, a feature compression module 706, an edge-end inference module 708, and an inference result fusion module 710 at the edge node, as well as a feature decompression module 712 and a cloud inference module 714 in the cloud.

[0111] The pre-inference module 702 mainly consists of convolutional neural network layers, which are responsible for extracting feature maps, and the results are used as input to the feature segmentation module.

[0112] The feature segmentation module 704 is used to sort the feature maps extracted by the "pre-inference module" in descending order according to the feature importance list obtained by the "feature importance estimation method", assign the "important features" with higher feature importance to the edge device for subsequent inference execution, and input the "remaining features" into the feature segmentation module 704.

[0113] The feature compression module 706 is used to compress the remaining features with lower importance and transmit them to the cloud device for further inference and prediction.

[0114] The feature decompression module 712 is used to decompress the compressed features sent from the edge using the same algorithm;

[0115] The edge inference module 708 is used to complete the features of important features through the random masking method. The completed feature data set is used for normal inference on the edge device, and the result is the input of the "inference result fusion module".

[0116] Cloud-based inference module 714: This module uses a random masking method to complete the remaining features. The completed feature data set is then used for normal inference on the cloud device, and the result is the input to the "inference result fusion module".

[0117] The inference result fusion module 710 is used to add the prediction results obtained from the edge and the cloud respectively to obtain the final inference result.

[0118] like Figure 8 As shown, the image prediction method based on an edge cloud architecture according to another embodiment of this disclosure specifically includes:

[0119] Step S802: For edge nodes with limited resources, the image to be predicted collected by the terminal device is input into the convolutional layer of the edge convolutional neural network for preprocessing to obtain a feature data set.

[0120] Step S804: By estimating the importance of the feature data, the feature data after layer processing of the edge convolutional neural network is sorted, and the important first group of features and the unimportant second group of features are obtained based on the sorting results.

[0121] Step S806: At the edge node, the second set of features is compressed and encoded by the compression module to reduce network traffic transmission and achieve reliable data transmission.

[0122] Step S808: The second set of features is decompressed in the cloud via the decompression module.

[0123] Step S810: At the edge nodes, feature completion is performed on the first set of features using a random masking model.

[0124] Step S812: In the cloud, feature completion is performed on the second set of features using a random masking model.

[0125] Step S814: Complete local inference at the edge nodes based on the first set of features completed.

[0126] Step S816: Cloud inference is completed on the cloud based on the completed second set of features.

[0127] Step S818: At the edge node, the inference results of the edge node are added and fused with the inference results of the cloud to obtain the final prediction result.

[0128] In this embodiment, through feature segmentation based on an edge-cloud architecture, important features are inferred at edge nodes, while unimportant features are inferred in the cloud, enhancing data security and improving the accuracy of inference on lightweight devices. Feature segmentation is performed based on feature importance estimation to achieve selective feature compression, reducing the accuracy drop caused by information loss during compression. A random masking mechanism is used to achieve the fusion of edge-cloud inference results, realizing the difference robustness of cloud-edge neural networks and improving inference accuracy.

[0129] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0130] The following reference Figure 9 To describe an image prediction device 900 based on an edge cloud architecture according to an embodiment of the present invention. Figure 9 The image prediction device 900 based on the edge cloud architecture shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0131] The image prediction device 900 based on an edge cloud architecture is manifested in the form of hardware modules. The components of the image prediction device 900 based on an edge cloud architecture may include, but are not limited to: a feature extraction module 902, used to perform feature extraction operations on the image to be predicted based on the feature extractor of the image prediction model in response to the acquired image to be predicted, generating a feature data set; a segmentation module 904, used to segment the feature data set into a first set of features and a second set of features based on an importance estimation operation; a first inference module 906, used to perform local inference operations on the first set of features, generating a first inference result; a first transmission module 908, used to send the second set of features to the cloud in the edge cloud architecture, so that the cloud performs cloud inference operations on the second set of features, generating a second inference result; and a fusion module 910, used to receive the second inference result sent from the cloud, and to fuse the first inference result and the second inference result to obtain the prediction result of the image to be predicted.

[0132] The following reference Figure 10 To describe an image prediction device 1000 based on an edge cloud architecture according to an embodiment of the present invention. Figure 10 The image prediction device 1000 based on the edge cloud architecture shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0133] The image prediction device 1000 based on an edge cloud architecture is manifested in the form of a hardware module. The components of the image prediction device 1000 based on an edge cloud architecture may include, but are not limited to: a receiving module 1002, used to receive a second set of features sent by edge nodes in the edge cloud architecture; a second inference operation module 1004, used to perform cloud inference operations on the second set of features to generate a second inference result; and a second sending module 1006, used to send the second inference result to the edge nodes.

[0134] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”

[0135] The following reference Figure 11 To describe an electronic device 1100 according to this embodiment of the present invention, it may be a network device or a terminal. Figure 11 The electronic device 1100 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0136] like Figure 11 As shown, the electronic device 1100 is manifested in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one storage unit 1120, and a bus 1130 connecting different system components (including storage unit 1120 and processing unit 1110).

[0137] The storage unit stores program code that can be executed by the processing unit 1110, causing the processing unit 1110 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1110 can perform actions such as... Figure 2 The scheme described in steps S202 to S211 shown.

[0138] Storage unit 1120 may include readable media in the form of volatile storage units, such as random access memory (RAM) 11201 and / or cache memory 11202, and may further include read-only memory (ROM) 11203.

[0139] Storage unit 1120 may also include a program / utility 11204 having a set (at least one) program module 11205, such program module 11205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0140] Bus 1130 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0141] Electronic device 1100 can also communicate with one or more external devices 1170 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1100, and / or any device that enables electronic device 1100 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1150. Furthermore, electronic device 1100 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1160. As shown, network adapter 1160 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0142] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0143] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on an electronic device, causes the electronic device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section above.

[0144] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on an electronic device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0145] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0146] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0147] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0148] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0149] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0150] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0151] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0152] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. An image prediction method based on an edge cloud architecture, characterized in that, Edge nodes applied in the aforementioned edge cloud architecture include: In response to the acquired image to be predicted, the feature extractor based on the image prediction model performs feature extraction operation on the image to be predicted to generate a feature data set; The feature data set is segmented into a first group of features and a second group of features based on an importance estimation operation, including: performing classification and recognition operations on the feature data set using a local classifier of the image prediction model to obtain type information of the image to be predicted; inputting the spatial feature information obtained by spatial feature mapping of the feature data and the type information into the interpretability model of the image prediction model to calculate the integral gradient of each feature data; performing fitting analysis on the integral gradient to obtain an importance distribution; wherein, if the importance distribution does not conform to the target importance distribution, calculating the deviation of the importance distribution relative to the target importance distribution based on the loss function of the image prediction model; optimizing the model parameters of the feature extractor and the local classifier based on the deviation to re-estimate the integral gradient as the importance of multiple feature data; sorting the multiple feature data from high to low importance; and segmenting the feature data set into a first group of features and a second group of features based on the sorting result. Perform local inference operations on the first set of features to generate a first inference result; The second set of features is sent to the cloud in the edge cloud architecture so that the cloud performs cloud inference operations on the second set of features to generate a second inference result; The system receives the second inference result sent from the cloud and performs a fusion operation on the first inference result and the second inference result to obtain the prediction result of the image to be predicted.

2. The image prediction method based on edge cloud architecture according to claim 1, characterized in that, The fitting analysis of the integral gradient of the multiple feature data includes: If the importance distribution is detected to conform to the target importance distribution, then the multiple integral gradients are determined as the importance of the multiple feature data. If the importance distribution is detected to be inconsistent with the target importance distribution, the deviation of the importance distribution relative to the target importance distribution is calculated based on the loss function of the image prediction model; The model parameters of the feature extractor and the local classifier are optimized based on the deviation, and the integral gradient of each feature data is updated based on the optimized feature extractor and the local classifier, so as to determine the importance of the multiple feature data by the updated integral gradient.

3. The image prediction method based on edge cloud architecture according to claim 1, characterized in that, The step of performing local inference operations on the first set of features to generate a first inference result includes: The first set of features is completed based on the random mask model and the number of features in the feature data set to obtain the first set of inference features; The first set of inference feature data is input into the local classifier to generate the first inference result based on the classification prediction of the local classifier.

4. The image prediction method based on edge cloud architecture according to claim 1, characterized in that, Before sending the second set of features to the cloud in the edge cloud architecture, the method further includes: The second set of features is compressed, and the compressed second set of features is sent to the cloud.

5. The image prediction method based on edge cloud architecture according to any one of claims 1 to 4, characterized in that, The step of fusing the first inference result and the second inference result to obtain the prediction result of the image to be predicted includes: A first weight value is determined based on the importance of the first set of features; The second weight value is determined based on the importance of the second set of features; The first inference result and the second inference result are added and fused based on the first weight value and the second weight value to obtain the prediction result.

6. An image prediction method based on an edge cloud architecture, characterized in that, Cloud applications in edge cloud architectures include: The system receives a second set of features sent by an edge node in the edge cloud architecture. The edge node segments the feature data set into a first set of features and a second set of features based on an importance estimation operation. The feature data set is generated by the edge node performing feature extraction on the acquired image to be predicted using a feature extractor based on an image prediction model. The segmentation of the feature data set into the first and second sets of features based on the importance estimation operation includes: performing classification and recognition operations on the feature data set using a local classifier based on the image prediction model to obtain the type information of the image to be predicted; and mapping the spatial feature information obtained by spatial feature mapping on the feature data and the... Type information is input into the interpretability model of the image prediction model to calculate the integral gradient of each feature data. The integral gradient is then fitted to obtain an importance distribution. If the importance distribution does not conform to the target importance distribution, the deviation of the importance distribution relative to the target importance distribution is calculated based on the loss function of the image prediction model. The model parameters of the feature extractor and the local classifier are optimized based on this deviation to re-estimate the integral gradient as the importance of multiple feature data. The importance of the multiple feature data is then sorted from high to low. Based on the sorting result, the feature data set is divided into a first group of features and a second group of features. Perform cloud-based inference operations on the second set of features to generate a second inference result; The second inference result is sent to the edge node.

7. The image prediction method based on edge cloud architecture according to claim 6, characterized in that, The second set of features received from the edge nodes in the edge cloud architecture includes: Receive compressed data sent by the edge node; The compressed data is decompressed to obtain the second set of features.

8. The image prediction method based on edge cloud architecture according to claim 6, characterized in that, The step of performing cloud-based inference operations on the second set of features to generate a second inference result includes: The second set of features is completed based on the random mask model and the number of features in the feature data set to obtain the second set of inference features; The second set of inference feature data is input into the cloud classifier to generate the second inference result based on the classification prediction of the cloud classifier.

9. An image prediction device based on an edge cloud architecture, characterized in that, Edge nodes applied in the aforementioned edge cloud architecture include: The feature extraction module is used to perform feature extraction operations on the image to be predicted based on the feature extractor of the image prediction model in response to the acquired image to be predicted, and generate a feature data set. A segmentation module is used to segment the feature data set into a first group of features and a second group of features based on an importance estimation operation. This includes: performing a classification and recognition operation on the feature data set using a local classifier of the image prediction model to obtain type information of the image to be predicted; inputting the spatial feature information obtained by spatial feature mapping of the feature data and the type information into the interpretability model of the image prediction model to calculate the integral gradient of each feature data; performing a fitting analysis on the integral gradient to obtain an importance distribution; wherein, if the importance distribution does not conform to the target importance distribution, calculating the deviation of the importance distribution relative to the target importance distribution based on the loss function of the image prediction model; optimizing the model parameters of the feature extractor and the local classifier based on the deviation to re-estimate the integral gradient as the importance of multiple feature data; sorting the importance of the multiple feature data from high to low; and segmenting the feature data set into a first group of features and a second group of features based on the sorting result. The first inference module is used to perform local inference operations on the first set of features and generate a first inference result. The first sending module is used to send the second set of features to the cloud in the edge cloud architecture, so that the cloud performs cloud inference operation on the second set of features and generates a second inference result; The fusion module is used to receive the second inference result sent from the cloud, and to perform a fusion operation on the first inference result and the second inference result to obtain the prediction result of the image to be predicted.

10. An image prediction device based on an edge cloud architecture, characterized in that, Cloud applications in edge cloud architectures include: The receiving module is configured to receive a second set of features sent by edge nodes in the edge cloud architecture. The edge nodes segment the feature data set into a first set of features and a second set of features based on an importance estimation operation. The feature data set is generated by the edge nodes performing feature extraction operations on the acquired image to be predicted using a feature extractor based on an image prediction model. The segmentation of the feature data set into the first and second sets of features based on the importance estimation operation includes: performing classification and recognition operations on the feature data set using a local classifier based on the image prediction model to obtain the type information of the image to be predicted; and performing spatial feature mapping on the feature data to obtain spatial feature information. The information and type information are input into the interpretability model of the image prediction model to calculate the integral gradient of each feature data. The integral gradient is then fitted to obtain an importance distribution. If the importance distribution does not conform to the target importance distribution, the deviation of the importance distribution relative to the target importance distribution is calculated based on the loss function of the image prediction model. The model parameters of the feature extractor and the local classifier are optimized based on the deviation to re-estimate the integral gradient as the importance of multiple feature data. The importance of the multiple feature data is sorted from high to low. Based on the sorting result, the feature data set is divided into a first group of features and a second group of features. The second inference operation module is used to perform cloud inference operations on the second set of features and generate a second inference result. The second sending module is used to send the second inference result to the edge node.

11. A network device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the image prediction method based on an edge cloud architecture according to any one of claims 1 to 5 or 6 to 8 by executing the executable instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image prediction method based on edge cloud architecture as described in any one of claims 1 to 8.

Images