A cloud-edge-device collaborative service matching method based on multimodal user preferences

Abstract

The present invention belongs to the field of service matching and deep learning technology, and discloses a cloud-edge collaborative service matching method based on multimodal user preferences. In the cloud-edge scenario, the present invention uses machine learning and deep learning models to perform service matching based on user short-term preference information and long-term service usage record data under the condition that the perception amount can be controlled. Among them, the user's multimodal information data is used to identify preference information using a recurrent neural network model and an attention mechanism as classifiers, and the K-nearest neighbor algorithm is used for the user's long-term access sequence to obtain the user's long-term preference. The present invention utilizes the characteristics of multimodal data and numerous method models of machine learning and deep learning to fully extract the large amount of multimodal data generated by users due to service calls, providing strong support for adaptive service matching in cloud-edge scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of service matching and deep learning technology, and specifically relates to a cloud-edge collaborative service matching method based on multimodal user preferences. Background Art

[0002] With the rapid development of the internet and computer technology, data and information are experiencing explosive growth. According to Cisco Cloud Forecast, by 2021, over 50 billion devices will be connected worldwide, generating over 84 zettabytes of data annually and placing a growing demand on high-performance computing. Furthermore, the expansion of IoT devices, the popularity of video content, and the influx of new users will drive a surge in IP traffic, projected to reach 4.8 zettabytes in 2022, more than triple the 2017 figure. Efficiently processing this massive amount of data and information has become a major challenge. Against this backdrop, cloud computing has emerged as a viable solution. Traditional cloud computing stores data, applications, and services in cloud centers, leveraging the powerful data storage and processing capabilities of cloud servers to provide abundant computing resources and deliver reliable services. However, due to the massive amounts of heterogeneous data generated by the growth of data-intensive and compute-intensive applications, cloud computing is no longer sufficient for processing latency-sensitive applications. Furthermore, uploading all data to the cloud significantly burdens network bandwidth. The emergence of edge computing (EC) allows users to avoid uploading their entire data to the cloud, instead processing it locally at nodes at the edge of the network. This strategy effectively reduces network bandwidth burdens and significantly reduces energy consumption and service latency for edge smart devices. Service matching is the process of finding and locating existing services based on the needs of service requesters. When registering services with a repository, service providers provide a number of similar functional descriptions, such as natural language-based service information, service categories, and service provider names. However, with the rapid development of internet technologies, the number of cloud-edge services is rapidly increasing. Accurately and efficiently discovering and matching services in cloud-edge-device scenarios has become a research hotspot in service computing. The emergence of multimodal learning technology offers a new approach to addressing this issue. Users in edge environments generate a large amount of data when invoking services. Fully extracting this multimodal data can provide powerful support for adaptive service matching. Therefore, it is crucial to study how to extract multimodal user data from edge environments, effectively represent this data, and solve service matching problems in cloud-edge-device scenarios. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention provides a cloud-edge collaborative service matching method based on multimodal user preferences. The method utilizes the characteristics of multimodal data and numerous method models of machine learning and deep learning to fully extract a large amount of multimodal data generated by users' service calls, providing strong support for adaptive service matching in cloud-edge scenarios.

[0004] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0005] The present invention is a cloud-edge collaborative service matching method based on multimodal user preferences. The cloud-edge collaborative service matching method includes a user preference information identification method integrating an attention mechanism and a user service usage record mining method based on K-nearest neighbors. Specifically,

[0006] The user information identification method integrating the attention mechanism is as follows: using the Text CNN model and the VGG16 model to mine user multimodal preferences, extracting text data and image data respectively, and using the attention mechanism-based classification network to jointly train the text data and image data. The Text CNN model and the VGG16 model identify the user's points of interest and deeply understand the contextual information, thereby improving the accuracy of service matching. The decision layer of the Text CNN model and the VGG16 model merges the classification results of the text data and image data, maintains the influence of different types of data on service selection, and derives a final comprehensive classification decision to obtain a short-term score S1(i);

[0007] The K-nearest neighbor-based user service usage record mining method is as follows: using the K-nearest neighbor sequence to analyze the user's long-term preferences, based on the user's previous browsing history, by calculating the ratio of the number of shared accessed services between the user's historical browsing history, that is, the user sequence and other sequences, to the total number of accessed services to measure similarity, find out the next service that the user may visit, and obtain a long-term score S2(i), and finally perform a weighted fusion of the vectors obtained by the two methods to obtain a comprehensive score of the service, and sort the services according to the comprehensive score to match the user with the top K services with the highest scores.

[0008] A further improvement of the present invention is that the user information recognition method integrating the attention mechanism specifically comprises the following steps:

[0009] Step 1.1, data acquisition and preprocessing: The data acquisition and preprocessing module is used to obtain and process data immediately, obtain text data and image data, and preprocess the obtained text data and image data;

[0010] Step 1.2: Feature extraction of text and image information: In text analysis, the Text CNN model is used to extract text data, and in image processing, the VGG16 model is used to refine raw image data.

[0011] Step 1.3: Process text and image data using the attention mechanism: Encode the context of text data and semantic content across different media, integrate and process data from the Text CNN model and the VGG16 model to obtain multimodal data;

[0012] Step 1.4: Fusion of preference features to determine classification results: Use a weighted fusion method to combine the multimodal data obtained in step 1.3, and multiply the user vector obtained after multimodal fusion with the service vector to obtain the short-term score S1(i) of the service.

[0013] A further improvement of the present invention is that: in step 1.1,

[0014] The preprocessing of text data specifically includes the following steps:

[0015] Step 1.1.1. First, delete the extra characters and punctuation marks in the text data, and split it into labeled words before feeding it into Text CNN.

[0016] Step 1.1.2: Convert the labeled words in step 1.1.1 into numerical vectors using the Glove model to capture the structure and meaning of the sentence.

[0017] The image data is preprocessed as follows: the image size is changed to 227×227 pixels, three channels (RGB) are maintained, and an image segmentation operation is performed.

[0018] A further improvement of the present invention is that the step 1.2 of using the VGG16 model to extract the original image features specifically includes the following steps:

[0019] Step 1.2.1: The original image first passes through the convolutional layers of the Text CNN model and the VGG16 model. The convolutional layers are connected to local areas of the image and extract image features through dot product operations.

[0020] In step 1.2.2, the convolutional layers of the Text CNN model and the VGG16 model are responsible for extracting image features, and the activation function ReLU normalizes the image size and enhances nonlinear characteristics;

[0021] Step 1.2.3: Set up a pooling layer to reduce the dimensionality of image features to reduce noise and improve computational efficiency.

[0022] In step 1.2.4, the image features are fed into the fully connected layers of the Text CNN model and the VGG16 model to complete the image feature processing.

[0023] A further improvement of the present invention is that: in step 1.3,

[0024] When processing text data through the attention mechanism, the initial step is to convert the input sentence into a vector form, and then multiply the generated vector with the corresponding attention weight to produce a weighted context vector;

[0025] When processing image data through the attention mechanism, the attention mechanism is used to identify and enhance key visual information and further locate the areas that attract user attention. Combined with the VGG16 model, the attention mechanism extracts important contextual features of the image by analyzing the correlation matrix of each spatial position within the feature map.

[0026] A further improvement of the present invention is that the method for mining user service usage records based on K nearest neighbors specifically includes the following steps:

[0027] Step 2.1: Based on the user's previous browsing history, find the next service the user may visit: Measure the similarity by calculating the ratio of the number of services visited in common between the user's browsing history, i.e., the user sequence and other sequences, to the total number of services visited. The formula is:

[0028] sim(s,n)=|s∩n| / |s∪n|

[0029] Where s represents the current user sequence, and n represents other sequences in text data and image data;

[0030] Step 2.2: Filter out the K sequences closest to the user sequence through similarity scoring to form the neighbor sequence set Ns of the current sequence;

[0031] Step 2.3: Assign a score to the service based on its frequency of appearance in the neighbor sequence set Ns, and obtain the long-term score S2(i) of the service;

[0032] Step 2.4: Combine the long-term score obtained in step 2.3 with the short-term score S1(i) to obtain the comprehensive score S(i) of the service;

[0033] Step 2.5: Sort by the comprehensive score S(i) and match the top K services with the highest scores to the user. The total service score S(i) is expressed as:

[0034] S(i)=S1(i)+S2(i).

[0035] The beneficial effects of the present invention are: the present invention provides a method for service matching based on machine learning and deep learning models under the condition that the perception amount can be controlled, using user short-term preference information and long-term service usage record data respectively. Through the user's multimodal information data, a recurrent neural network model and an attention mechanism are used as classifiers to identify preference information, and the K-nearest neighbor algorithm is used for the user's long-term access sequence to obtain the user's long-term preference.

[0036] This invention utilizes the characteristics of multimodal data and numerous methods and models of machine learning and deep learning to fully extract the large amount of multimodal data generated by users' service calls, providing strong support for adaptive matching of services in cloud-edge scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a basic research framework diagram of the present invention.

[0038] Figure 2 This is a basic process framework diagram for identifying user preference information based on recurrent neural networks and attention mechanisms in the present invention.

[0039] Figure 3 This is a basic process framework diagram for mining users' long-term service usage records based on the K-nearest neighbor algorithm in the present invention. DETAILED DESCRIPTION

[0040] The following diagrams illustrate embodiments of the present invention. For clarity, many practical details are included in the following description. However, it should be understood that these practical details are not intended to limit the present invention. In other words, in some embodiments of the present invention, these practical details are not essential.

[0041] Example 1

[0042] like Figure 1 As shown, the present invention is a cloud-edge collaborative service matching method based on multimodal user preferences, which includes a user preference information identification method integrating an attention mechanism and a user service usage record mining method based on K-nearest neighbors:

[0043] The user information identification method integrating the attention mechanism is as follows: using the Text CNN model and the VGG16 model to mine user multimodal preferences, extracting text data and image data respectively, and based on the classification network of the attention mechanism, through joint training of text data and image data, the Text CNN model and the VGG16 model can more accurately identify the user's points of interest and deeply understand the contextual information, thereby improving the accuracy of service matching. Since text and image features are significantly different in nature, the classification results of the text data and image data are merged at the decision layer of the Text CNN model and the VGG16 model, aiming to maintain the influence of different types of data on service selection to the greatest extent, and to obtain the final comprehensive classification decision to obtain the short-term score S1(i).

[0044] The K-nearest neighbor-based user service usage record mining method is as follows: using the K-nearest neighbor sequence to analyze the user's long-term preferences, based on the user's previous browsing history, by calculating the ratio of the number of shared accessed services between the user's historical browsing history, that is, the user sequence and other sequences, to the total number of accessed services to measure similarity, find out the next service the user may visit, and obtain a long-term score S2(i). Finally, the vectors obtained by the two methods are weightedly fused to achieve a balance between system power consumption and performance, remove bias information in the sequence to a certain extent, make the preference information more concentrated, and match the user with the service that best meets their needs.

[0045] like Figure 2 As shown in FIG, the user information recognition method integrating the attention mechanism specifically includes the following steps:

[0046] Step 1.1: Data collection and preprocessing.

[0047] Deploy data collection and preprocessing modules at the data acquisition terminal to realize real-time data acquisition and processing; the collected data should include both text data and image data.

[0048] The preprocessing of text data is as follows: first, redundant characters and punctuation marks are deleted from the text data. Before the text data is input into Text CNN, it is first split into labeled words. Through the Glove model, these words are converted into numerical vectors to capture the structure and meaning of the sentence.

[0049] The image data is preprocessed by resizing it to 227×227 pixels, maintaining three channels (RGB), and performing image segmentation. To maintain consistency, the segments overlap to a certain extent, which avoids misidentification of event types that might otherwise occur due to undersized segments.

[0050] Step 1.2: Feature Extraction of Text and Image Information: For text analysis, the Text CNN model is used to extract text data. The main advantage of this convolutional neural network architecture is that it requires fewer parameter updates, which improves processing speed and makes the model more efficient and faster during iteration than other traditional models. For image processing, the VGG16 model is used to refine the raw image data. The image first passes through a series of convolutional layers, which connect to local regions of the image and extract image features through dot product operations. These operations are performed using a pre-trained VGG16 network. The convolutional layers extract features, and the ReLU activation function normalizes the image size and enhances nonlinear characteristics. Pooling layers are then used to reduce the dimensionality of the features, reducing noise and improving computational efficiency. Finally, these features are fed into a fully connected layer to complete the feature processing.

[0051] Step 1.3: Process text and image data through the attention mechanism: Encode the context of text data and semantic content across different media, integrate and process data from the Text CNN model and the VGG16 model to obtain multimodal data.

[0052] The attention mechanism has the ability to filter key components of an input sequence by assigning varying levels of importance to different parts of multimodal information. This mechanism extracts important information from hidden states, from the word level to the sentence level, and leverages this information for service matching tasks. When applied to text and image data, the attention mechanism encodes the context and semantic content of the text across different media, integrating and processing information from multiple models. Specific operations tailored to the characteristics of different modal data are incorporated into the encoding, decoding, and self-attention processes, making it suitable for multimodal applications. When processing text data, the initial step is to convert the input sentence into a vector. These vectors are then multiplied by corresponding attention weights to produce a weighted context vector. This entire process, through the attention mechanism, strengthens the model's focus on key words in the sentence. Within this framework, each hidden layer of the model incorporates information from the entire input sequence to capture the global context of the sentence. When processing image data, the attention mechanism is used to identify and enhance key visual information, further localizing areas of user attention. In conjunction with the VGG16 architecture, this mechanism effectively extracts important contextual features of the image by analyzing the correlation matrix of spatial locations within the feature map.

[0053] Step 1.4: Fusion of preference features to determine classification results: When performing service matching tasks, the contributions of various modalities may vary. Therefore, before performing modal fusion, it is necessary to assess the relative importance of each modality. A weighted fusion approach can effectively combine data from different modalities. This process involves assigning a weight to each modality, which reflects the weight it plays in the final classification decision. This ensures that, during feature fusion, each modality contributes appropriately to the classification result based on its impact. Multiplying the user vector obtained by fusing the modalities with the service vector yields the service's recent score, S1(i).

[0054] like Figure 3 As shown, the user service usage record mining method based on K nearest neighbors.

[0055] Traditional service matching algorithms primarily focus on leveraging users' personal preferences to recommend services. This approach often overlooks a crucial element contained in user historical data: the synergistic effects of other user behavior sequences that are similar to a specific user's historical behavior patterns. These similar sequences, referred to as "adjacent sequences," not only contain behavioral patterns similar to the current user sequence but also reflect similar user intent. The commonalities between these adjacent sequences, or their shared behavioral characteristics, are actually a form of potential collaborative information that algorithms can leverage to more accurately predict and match services, thereby improving the relevance and personalization of service recommendations. Therefore, integrating this collaborative information into the service matching process can significantly enhance the intelligence of the matching system, enabling it to not only rely solely on the preferences of individual users but also mine and analyze behavioral correlations across multiple users to achieve more accurate and efficient service matching.

[0056] The K-nearest neighbor sequence is used to analyze the user's long-term preferences. This method finds the next service that the user may visit based on the user's previous browsing history. Specifically, the similarity is measured by calculating the ratio of the number of common visited services to the total number of visited services between the user's historical browsing history (sequence) and other sequences. The calculation is performed using the formula sim(s, n) = |s∩n| / |s∪n|, where s represents the current user sequence and n represents other sequences in the data set. Through similarity scoring, the K sequences closest to the user sequence are screened to form the neighbor sequence set Ns of the current sequence; next, the service is assigned a score based on the frequency of these services appearing in the neighbor sequence set to obtain the long-term score S2(i) of the service. This long-term score is combined with the short-term score S1(i) to obtain the comprehensive score S(i) of the service; finally, the services are sorted according to the comprehensive score S(i), and the top K services with the highest scores are matched to the user. The expression of the total service score S(i) is:

[0057] S(i)=S1(i)+S2(i)

[0058] Example 2

[0059] This embodiment provides an electronic device comprising at least one processing unit and at least one storage medium that can be read by the processing unit, wherein the storage medium contains at least one program. When this program or multiple programs are running, it guides at least one processing unit to execute a cloud-edge collaborative service matching method based on multimodal user preferences.

[0060] Example 2

[0061] This embodiment provides a computer-readable storage medium, characterized in that: the storage medium is equipped with codes executed by a computer. When these codes are executed, a cloud-edge collaborative service matching method based on multimodal user preferences is implemented.

[0062] The present invention utilizes the following principles: The present invention is mainly aimed at users in cloud-edge scenarios. Based on deep learning technology, it uses multimodal user data to study service matching methods in cloud-edge scenarios. Different research methods and mechanisms are adopted in two different machine learning and deep learning architectures. In the framework of recurrent neural networks, the attention mechanism is used to assign different importance to each part of the multimodal information, extract important information in the hidden state from the word level to the sentence level, and use this information to perform service matching tasks; in the framework of the K-nearest neighbor algorithm, the K-nearest neighbor sequence of the user's long-term access sequence is used to obtain the user's long-term preferences.

[0063] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A cloud-edge-device collaborative service matching method based on multimodal user preferences, characterized by: The cloud-edge collaborative service matching method includes a user preference information identification method integrating an attention mechanism and a user service usage record mining method based on K-nearest neighbors. Specifically, The user information identification method integrating the attention mechanism is as follows: using the Text CNN model and the VGG16 model to mine user multimodal preferences, extracting text data and image data respectively, and using the attention mechanism-based classification network to jointly train the text data and image data. The Text CNN model and the VGG16 model identify the user's points of interest and deeply understand the contextual information, thereby improving the accuracy of service matching. The decision layer of the Text CNN model and the VGG16 model merges the classification results of the text data and image data, maintains the influence of different types of data on service selection, and derives a final comprehensive classification decision to obtain a short-term score S1(i); The K-nearest neighbor-based user service usage record mining method is as follows: using the K-nearest neighbor sequence to analyze the user's long-term preferences, based on the user's previous browsing history, by calculating the ratio of the number of shared accessed services between the user's historical browsing history, i.e., the user sequence and other sequences, to the total number of accessed services to measure similarity, find the next service the user is likely to access, and obtain a long-term score S2(i). Finally, the vectors obtained by the two methods are weighted and fused to obtain a comprehensive score of the service, and the top K services with the highest scores are sorted according to the comprehensive scores to match the user; wherein: The user information recognition method integrating the attention mechanism specifically comprises the following steps: Step 1.1, data acquisition and preprocessing: The data acquisition and preprocessing module is used to obtain and process data immediately, obtain text data and image data, and preprocess the obtained text data and image data; Step 1.2: Feature extraction of text and image information: In text analysis, the Text CNN model is used to extract text data, and in image processing, the VGG16 model is used to refine raw image data. Step 1.3: Process text and image data using the attention mechanism: Encode the context of text data and semantic content across different media, integrate and process data from the Text CNN model and the VGG16 model to obtain multimodal data; Step 1.4: Fusion of preference features to determine classification results: Use a weighted fusion method to combine the multimodal data obtained in step 1.

3. Multiply the user vector obtained after multimodal fusion with the service vector to obtain the short-term score S1(i) of the service. The step 1.2 of using the VGG16 model to extract the original image features specifically includes the following steps: Step 1.2.1: The original image first passes through the convolutional layers of the Text CNN model and the VGG16 model. The convolutional layers are connected to local areas of the image and extract image features through dot product operations. In step 1.2.2, the convolutional layers of the Text CNN model and the VGG16 model are responsible for extracting image features, and the activation function ReLU normalizes the image size and enhances nonlinear characteristics; Step 1.2.3: Set up a pooling layer to reduce the dimensionality of image features to reduce noise and improve computational efficiency. In step 1.2.4, the image features are fed into the fully connected layers of the Text CNN model and the VGG16 model to complete the image feature processing. In step 1.3, when processing text data through the attention mechanism, the initial step is to convert the input sentence into a vector form, and then multiply the generated vector by the corresponding attention weight to produce a weighted context vector. When processing image data through the attention mechanism, the attention mechanism is used to identify and enhance key visual information and further locate the areas that attract user attention. In combination with the VGG16 model, the attention mechanism extracts important context features of the image by analyzing the correlation matrix of each spatial position within the feature map. The K-nearest neighbor-based user service usage record mining method specifically includes the following steps: Step 2.1: Based on the user's previous browsing history, find the next service the user may visit: Measure the similarity by calculating the ratio of the number of services visited in common between the user's browsing history, i.e., the user sequence and other sequences, to the total number of services visited. The formula is: sim(s,n)=|s∩n| / |s∪n| Where s represents the current user sequence, and n represents other sequences in text data and image data; Step 2.2: Filter out the K sequences closest to the user sequence through similarity scoring to form the neighbor sequence set Ns of the current sequence; Step 2.3: Assign a score to the service based on its frequency of appearance in the neighbor sequence set Ns, and obtain the long-term score S2(i) of the service; Step 2.4: Combine the long-term score obtained in step 2.3 with the short-term score S1(i) to obtain the comprehensive score S(i) of the service; Step 2.5: Sort by the comprehensive score S(i) and match the top K services with the highest scores to the user. The total service score S(i) is expressed as: S(i)=S1(i)+S2(i).

2. The cloud-edge-device collaborative service matching method based on multimodal user preferences according to claim 1 is characterized by: In step 1.1, The preprocessing of text data specifically includes the following steps: Step 1.1.

1. First, delete the extra characters and punctuation marks in the text data, and split it into labeled words before feeding it into Text CNN. Step 1.1.2: Convert the labeled words in step 1.1.1 into numerical vectors using the Glove model to capture the structure and meaning of the sentence. The image data is preprocessed as follows: the image size is changed to 227×227 pixels, three channels (RGB) are maintained, and an image segmentation operation is performed.

3. An electronic device comprising at least one processing unit and at least one storage medium readable by the processing unit, characterized in that: The storage medium contains at least one program. When this program or multiple programs are running, it guides at least one processing unit to execute a cloud-edge collaborative service matching method based on multimodal user preferences as described in any one of claims 1-2.

4. A computer-readable storage medium, characterized in that: The storage medium carries codes executed by a computer. When these codes are executed, a cloud-edge collaborative service matching method based on multimodal user preferences as described in any one of claims 1-2 is implemented.

Images