Cooperative working method based on cloud computer and related equipment

Abstract

The invention provides a cloud computer-based cooperative working method, which comprises the following steps of: acquiring a working interface image of any target user in a plurality of clients, and generating a working interface image sequence; performing semantic feature extraction on the work interface image sequence, and determining a current task type corresponding to the target user; calculating an interface change ratio of the working interface image sequence; selecting a candidate processing channel set from a predefined processing channel resource pool according to the current task type and the interface change ratio; acquiring load data of the cloud computer system in real time, and predicting a load state of each processing channel in a preset time period; determining a target processing channel in the candidate channel set based on the load state; and performing cloud data processing on the working interface image sequence of the target user according to the target processing channel to generate a cloud processing result, and performing cooperative processing on the work of the target user according to the cloud processing result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computers, and in particular to a collaborative working method based on a cloud computer and related equipment. BACKGROUND

[0002] Cloud computer technology provides flexible remote desktop services for users by concentrating computing and storage resources in the cloud. In a multi-user collaborative office scenario, multiple users need to access the same cloud desktop environment or share applications through their respective clients to jointly complete a task. Currently, existing collaborative methods mainly allocate resources according to the interface change rate of a single user, without considering task dependencies, resulting in resource allocation conflicts or delays in critical tasks. They rely on historical latency records for prediction, without combining real-time task status and future load changes, resulting in resource allocation conflicts or delays in critical tasks. Therefore, the existing collaborative working method has the problem of optimizing the behavior of a single user, without considering task dependencies, and relying on historical latency records for prediction, without combining real-time task status and future load changes, resulting in resource allocation conflicts or delays in critical tasks. SUMMARY

[0003] The present application provides a collaborative working method based on a cloud computer, which aims to solve the problem of the existing collaborative working method of optimizing the behavior of a single user, relying on historical latency records for prediction, and not combining real-time task status and future load changes, resulting in resource allocation conflicts or delays in critical tasks. By extracting semantic features from the work interface image sequence of any target user in multiple clients, the current task type corresponding to the target user is determined, the interface change rate of the work interface image sequence is calculated, and a candidate processing channel set is selected from a predefined processing channel resource pool according to the current task type and the interface change rate. The load data of the cloud computer system is collected in real time, and the load status of each processing channel in a preset time period is predicted. According to the load status, the target processing channel is determined in the candidate channel set. The work interface image sequence of the target user is processed by the target processing channel, and the cloud processing result is generated. The work of the target user is collaboratively processed according to the cloud processing result, solving the problem of the existing collaborative working method of optimizing the behavior of a single user, relying on historical latency records for prediction, and not combining real-time task status and future load changes, resulting in resource allocation conflicts or delays in critical tasks.

[0004] In a first aspect, the present application provides a collaborative working method based on a cloud computer, which includes the following steps: Collecting the work interface image of any target user in multiple clients and generating a work interface image sequence; performing semantic feature extraction on the working interface image sequence to determine a current task type corresponding to the target user; calculating an interface change ratio of the working interface image sequence; selecting a candidate processing channel set from a predefined processing channel resource pool according to the current task type and the interface change ratio; collecting load data of the cloud computer system in real time, and predicting load states of each processing channel in a preset time period; determining a target processing channel from the candidate channel set based on the load states; performing cloud data processing on the working interface image sequence of the target user according to the target processing channel, generating a cloud processing result, and performing collaborative processing on the work of the target user according to the cloud processing result.

[0005] Optionally, the performing semantic feature extraction on the working interface image sequence to determine a current task type corresponding to the target user comprises: performing region segmentation and semantic recognition on each working interface image in the working interface image sequence through a trained task type classification model to generate corresponding semantic labels; monitoring user operation behaviors of each working interface image in the working interface image sequence to generate user operation logs; obtaining the current task type corresponding to the target user based on the semantic labels and the user operation logs, wherein the trained task type classification model is obtained by training a pre-trained task type classification model through a training data set, the training data set comprises sample working interface image data, semantic annotation data and user operation annotation data of the sample working interface image data, and the pre-trained task type classification model outputs a task type of the sample working interface image data.

[0006] Optionally, the calculating an interface change ratio of the working interface image sequence comprises: extracting a first interface image and a second interface image of adjacent images from the working interface image sequence; calculating a color feature deviation rate, a texture feature deviation rate and a semantic feature deviation rate between the first interface image and the second interface image; performing weighted fusion processing on the color feature deviation rate, the texture feature deviation rate and the semantic feature deviation rate to obtain the interface change ratio of the working interface image sequence.

[0007] Optionally, the calculating a color feature deviation rate and a texture feature deviation rate between the first interface image and the second interface image comprises: performing first image feature extraction processing on the first interface image to obtain first image features corresponding to the first interface image; performing second image feature extraction processing on the second interface image to obtain second image features corresponding to the second interface image; calculating a difference degree between the first image features and the second image features in a color space to obtain a color feature deviation rate; calculating a difference degree between the first image features and the second image features in a texture structure to obtain a texture feature deviation rate.

[0008] Optionally, the calculating of the semantic feature deviation rate between the first interface image and the second interface image comprises: performing first semantic recognition on the first interface image to obtain first semantic feature vectors corresponding to the first interface image; performing second semantic recognition on the second interface image to obtain second semantic feature vectors corresponding to the second interface image; calculating a feature distance between the first semantic feature vectors and the second semantic feature vectors to obtain a semantic feature deviation rate.

[0009] Optionally, the real-time collection of the load data of the cloud computer system and the prediction of the load states of each processing channel in a preset time period comprise: real-time collection of historical and real-time load data of other users in the cloud computer system who are in the same collaborative session as the target user; inputting the load data into a preset load prediction model to predict load states of each processing channel in a preset time period in a next working stage of the collaborative session, wherein the preset load prediction model is obtained by training a pre-trained load prediction model by using a load training data set, and the load training data set comprises historical load data and corresponding historical load state label data, real-time load data and corresponding real-time load state label data.

[0010] Optionally, the determining of the target processing channel from the candidate processing channel set based on the load state comprises: calculating a corresponding comprehensive optimization score for each candidate processing channel in the candidate processing channel set based on a multi-objective optimization algorithm; selecting, according to the load state, a candidate processing channel with the highest comprehensive optimization score from the candidate processing channel set as the target processing channel.

[0011] In a second aspect, an embodiment of the present application further provides a collaborative working device based on a cloud computer, which comprises: The generating module is configured to collect a work interface image of any target user in the plurality of clients and generate a work interface image sequence; The first determining module is configured to perform semantic feature extraction on the work interface image sequence and determine a current task type corresponding to the target user; The calculating module is configured to calculate an interface change ratio of the work interface image sequence; The selecting module is configured to select a candidate processing channel set from a predefined processing channel resource pool according to the current task type and the interface change ratio; The predicting module is configured to collect load data of the cloud computer system in real time and predict load states of each processing channel in a preset time period; The second determining module is configured to determine a target processing channel from the candidate channel set based on the load states; The cooperative processing module is configured to perform cloud data processing on the work interface image sequence of the target user according to the target processing channel, generate a cloud processing result, and perform cooperative processing on the work of the target user according to the cloud processing result.

[0012] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the cloud computer-based cooperative work method provided by the embodiments of the present application when executing the computer program. In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps in the cloud computer-based cooperative work method provided by the embodiments of the present application when executed by a processor.

[0013] In the embodiment of the present application, the working interface image of any target user in the plurality of clients is collected, and a working interface image sequence is generated; the semantic feature of the working interface image sequence is extracted to determine the current task type corresponding to the target user; the interface change ratio of the working interface image sequence is calculated; the candidate processing channel set is selected from the predefined processing channel resource pool according to the current task type and the interface change ratio; the load data of the cloud computer system is collected in real time, and the load state of each processing channel in the preset time period is predicted; the target processing channel is determined in the candidate channel set based on the load state; the cloud data processing is performed on the working interface image sequence of the target user according to the target processing channel, the cloud processing result is generated, and the work of the target user is cooperatively processed according to the cloud processing result. The present application extracts the semantic feature of the working interface image sequence of any target user in the plurality of clients, determines the current task type corresponding to the target user, calculates the interface change ratio of the working interface image sequence, selects the candidate processing channel set from the predefined processing channel resource pool according to the current task type and the interface change ratio, collects the load data of the cloud computer system in real time, predicts the load state of each processing channel in the preset time period, determines the target processing channel in the candidate channel set according to the load state, performs cloud data processing on the working interface image sequence of the target user according to the target processing channel, generates the cloud processing result, and cooperatively processes the work of the target user according to the cloud processing result. The problem of resource allocation conflict or key task delay caused by the fact that the existing collaborative work method optimizes the behavior of a single user, does not consider task dependency, and relies on historical time delay records for prediction without considering real-time task state and future load change is solved. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0015] Figure 1 is a flowchart of a collaborative work method based on a cloud computer provided by the embodiment of the present application; Figure 2 is a structural schematic diagram of a collaborative work device based on a cloud computer provided by the embodiment of the present application; Figure 3 is a structural schematic diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0016] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0017] As shown in Figure 1 , Figure 1 is a flowchart of a cloud computer-based collaborative work method provided by an embodiment of the present application. The cloud computer-based collaborative work method comprises the following steps: 101. Collecting a work interface image of any target user in a plurality of clients, and generating a work interface image sequence.

[0018] In the embodiments of the present application, the cloud computer-based collaborative work method can be applied to a cloud computer. The cloud computer is a virtual desktop service based on cloud computing technology, which migrates the computing, storage and running environment of a traditional personal computer to a cloud server. A user can access a cloud computer with complete functions through a light terminal device (such as a notebook, a tablet or a mobile phone) connected to a network. The cloud computer-based collaborative work can be applied to a collaborative work system implemented by the cloud computer. The collaborative work system comprises a plurality of clients, and the plurality of clients are used for collaborative processing of a target work. The collaborative processing can be understood as a process in which the plurality of clients complete the work by information exchange and resource sharing under a common target.

[0019] The target user can be any user in the plurality of clients.

[0020] The work interface image can be understood as an image displayed on a screen in a cloud computer work environment. The work interface image contains elements such as documents, charts and web pages. For example, the work interface image of a designer can contain the interface of a design software and a project file being processed.

[0021] The generation can be understood as a process of generating a work interface image sequence in chronological order from the work interface image of the target user.

[0022] The work interface image sequence can be a continuous work interface image arranged in chronological order from the work interface image of the target user.

[0023] It should be noted that the working interface image of any target user in the plurality of clients can be captured in real time using screen capture software or API, and then the working interface images are sorted in time sequence to obtain a working interface image sequence. For example, when user A is processing a file, browsing a webpage, or performing other tasks, the working interface image of user A can be captured and saved as a working interface image sequence.

[0024] 102. Perform semantic feature extraction on the working interface image sequence to determine the current task type corresponding to the target user.

[0025] In the embodiments of the present application, the above-mentioned semantic extraction process can be understood as identifying the process that can reflect the content, meaning or purpose of the working interface image in each working interface image in the working interface image sequence.

[0026] Further, the semantic feature extraction process can be performed on each working interface image in the working interface image sequence by a task recognition model to obtain the semantic features corresponding to the working interface image, and the corresponding current task type is determined according to the semantic features. The above-mentioned task recognition model can be a task recognition model constructed based on deep learning or machine learning, such as a Transformer model, CNN, etc. The above-mentioned Transformer model is a deep learning model based on self-attention mechanism, which is used to process sequence data such as text, image or video, etc. The Transformer model can process information in parallel and capture long-distance dependencies. The above-mentioned CNN (Convolutional Neural Network) can automatically extract local features of the image and classify by simulating the working principle of the human visual system.

[0027] The above-mentioned current task type can be the task type that the target user is currently performing, such as document editing, data analysis, etc., which is obtained by real-time identification and classification according to the specific operation or matter that the target user is currently performing on the working interface image.

[0028] In a possible embodiment, for example, the working interface image of the user is identified as the text content input in the office software document, the position and operation mode of the mouse click, etc., and it can be determined that the current task type of the user is a report writing, data analysis, etc.

[0029] 103. Calculate the interface change rate of the working interface image sequence.

[0030] In the embodiments of the present application, the above-mentioned interface change rate can be understood as the change degree between adjacent two images in the working interface image sequence, which can measure the change degree of the working interface sequence by calculating the pixel difference or feature difference of the image.

[0031] Further, an optical flow method, a SIFT algorithm, etc. can be used to calculate the change degree between two adjacent images in the working interface image sequence. The optical flow method is an image processing technique for estimating the motion speed of a pixel point in an image. The optical flow method infers the moving direction and speed of an object by calculating the brightness change between consecutive frames. The SIFT algorithm (Scale Invariant Feature Transform) is a computer vision algorithm for detecting and describing local features of an image. The core of the SIFT algorithm is to extract key points and descriptors in an image that are invariant to scale, rotation, and affine transformation by constructing a scale space.

[0032] In a possible embodiment, the optical flow method can be used to calculate the motion vector of an image frame by frame, and the motion distance of each pixel is counted. Then, the average motion distance of the entire image can be calculated, and the average motion distance is taken as the interface change ratio of the frame. Specifically, for each image frame A, the motion vector field of the image frame A can be calculated using the optical flow method, and the motion vector field of the image frame A includes the motion direction and speed of each pixel in the image. For each pixel, the motion distance of each pixel in all frames is counted, the average motion distance of the entire image is calculated, and the average motion distance is obtained by averaging the motion distances of all pixels, and the average motion distance is taken as the interface change ratio of the image frame A. For example, there is an image sequence with a size of 100x100 pixels, and the motion vector field of each image frame B has been calculated. The motion distance of each pixel in all frames can be found out. For example, pixel C moves 50 pixels in 5 frames, so the motion distance of pixel C is 50. Then, the average value of the motion distances of all pixels can be calculated. For example, the average value is 20, and 20 is taken as the interface change ratio of the image frame B.

[0033] 104. Selecting a candidate processing channel set from a predefined processing channel resource pool according to the current task type and the interface change ratio.

[0034] In the embodiment of the present application, the predefined processing channel resource pool can be a predefined processing channel set. Different task types correspond to different performance processing channels, and a series of heterogeneous and available virtualization processing channels can be divided and encapsulated according to different data processing strategies and hardware performance. For example, different task types correspond to different algorithms, services, or programs, and different interface change ratios correspond to different algorithms, services, or programs.

[0035] Further, a plurality of candidate processing channels that are most suitable for processing the current task and the interface change ratio can be selected as the candidate processing channel set from the predefined processing channel resource pool according to the current task type and the interface change ratio.

[0036] The candidate processing channel can be a candidate processing channel.

[0037] The candidate processing channel set can be a priority queue set for classifying and scheduling different service flows.

[0038] It should be noted that the predefined processing channel resource pool can be classified and labeled according to the task type and the interface change rate, and the most matched candidate processing channel set can be selected from the predefined processing channel resource pool according to the current task type and the interface change rate.

[0039] In a possible embodiment, for example, the predefined processing channel resource pool has channel A, channel B, and channel C, wherein channel A is suitable for high-priority tasks, channel B is suitable for medium-priority tasks, and channel C is suitable for low-priority tasks. When a new task needs to be processed, the most matched candidate processing channel set can be selected from the predefined processing channel resource pool according to the current task type and the interface change rate. When the current task type is high-priority and the interface change rate is high, channel A is selected as the channel for processing the current task; when the current task type is medium-priority and the interface change rate is moderate, channel B can be selected as the channel for processing the current task; and when the current task type is low-priority and the interface change rate is low, channel C can be selected as the channel for processing the current task, which helps to improve the efficiency and quality of task processing.

[0040] 105. Collect load data of the cloud computer system in real time, and predict load states of each processing channel in a preset time period.

[0041] In the embodiment of the present application, the load data can be the workload of the cloud computer system in processing tasks within the working time, and the load data includes the usage of CPU, memory, and disk.

[0042] Further, the load states of each processing channel in the preset time period can be predicted by a load prediction model. The load prediction model can be a load prediction model constructed based on deep learning or machine learning, such as an LSTM neural network, a support vector machine (SVM), etc. The LSTM neural network (Long Short-Term Memory Network) is a time recurrent neural network specially designed to solve the long-term dependence problem of recurrent neural network (RNN), which controls the storage and update of information by introducing a "gate mechanism". The support vector machine (SVM) is a supervised learning algorithm for binary classification, and the core idea of the support vector machine is to maximize the classification interval by finding an optimal hyperplane, thereby achieving efficient classification of data.

[0043] The preset time period can be understood as a future time period set in advance, such as 5 seconds, 8 seconds, 10 seconds, etc.

[0044] The load status can be a load status of each processing channel in a preset time period, such as usage of CPU, memory, and the like.

[0045] 106. Based on the load status, a target processing channel is determined from the candidate channel set.

[0046] In the embodiments of the present application, the optimal processing channel can be selected from the candidate channel set as the target processing channel according to the load status of each processing channel.

[0047] It should be noted that the performance of each candidate channel, such as transmission speed, stability, and the like, can be evaluated according to the load status of each processing channel, and the candidate processing channel with the optimal performance or best suited to the current load status can be selected from the candidate channel set as the target processing channel.

[0048] 107. The cloud data processing of the target user's work interface image sequence is performed according to the target processing channel, a cloud processing result is generated, and the work of the target user is collaboratively processed according to the cloud processing result.

[0049] In the embodiments of the present application, the cloud data processing can be understood as a processing process operation of performing calculation and rendering of the target user's work interface image sequence according to the target processing channel in the cloud computer collaborative office scenario.

[0050] The cloud processing result can be understood as a data packet or data stream that can be directly received and used by the client after the cloud server completes logical calculation and graphic rendering of the target user's work interface image sequence according to the target processing channel.

[0051] The collaborative processing can be understood as a process and state in which multiple users based on the cloud computer architecture share the same work environment, data, or application, and synchronously or asynchronously, efficiently and smoothly complete a common work goal by using the cloud processing result.

[0052] It should be noted that the cloud data processing of the work interface image sequence is performed through the target processing channel, a cloud processing result is generated, and the work is collaboratively processed according to the cloud processing result, which can improve work efficiency.

[0053] In the embodiment of the present application, the working interface image of any target user in the plurality of clients is collected, and a working interface image sequence is generated; the semantic feature of the working interface image sequence is extracted to determine the current task type corresponding to the target user; the interface change rate of the working interface image sequence is calculated; according to the current task type and the interface change rate, a candidate processing channel set is selected from a predefined processing channel resource pool; the load data of the cloud computer system is collected in real time, and the load state of each processing channel in a preset time period is predicted; based on the load state, the target processing channel is determined in the candidate channel set; the working interface image sequence of the target user is processed by the target processing channel to generate a cloud processing result, and the work of the target user is cooperatively processed according to the cloud processing result. The present application extracts the semantic features of the working interface image sequence of any target user in the plurality of clients, determines the current task type corresponding to the target user, calculates the interface change rate of the working interface image sequence, selects a candidate processing channel set from a predefined processing channel resource pool according to the current task type and the interface change rate, collects the load data of the cloud computer system in real time, predicts the load state of each processing channel in a preset time period, determines the target processing channel in the candidate channel set according to the load state, processes the working interface image sequence of the target user according to the target processing channel to generate a cloud processing result, and cooperatively processes the work of the target user according to the cloud processing result. The present application solves the problem that the existing collaborative work method optimizes the behavior of a single user, does not consider task dependency, and relies on historical delay records for prediction, without considering real-time task state and future load changes, resulting in resource allocation conflicts or key task delays.

[0054] It can be understood that in the specific embodiments of the present application, image data, semantic data, feature data, load data, user data and other related data are involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data, as well as the training, deployment and calling of algorithm models, need to comply with relevant laws, regulations and standards of the country and region.

[0055] Optionally, in the step of extracting the semantic features of the working interface image sequence to determine the current task type corresponding to the target user, the trained task type classification model can be used to perform region segmentation and semantic recognition on each working interface image in the working interface image sequence to generate corresponding semantic labels; the user operation behavior of each working interface image in the working interface image sequence is monitored to generate a user operation log; based on the semantic labels and the user operation log, the current task type corresponding to the target user is obtained.

[0056] In the embodiments of the present application, the trained task type classification model is obtained by training a pre-trained task type classification model based on a training data set. The training data set includes sample work interface image data, semantic annotation data of the sample work interface image data, and user operation annotation data. The pre-trained task type classification model outputs the task type of the sample work interface image data. The annotation data is a process of converting original data (such as images, videos, etc.) into structured information recognizable by machine learning algorithms by adding labels, annotations or markers. Through the annotated data, the machine can learn the characteristics of the data, so as to improve the accuracy and efficiency of the algorithm. The above training can be supervised training. The supervised training is a process of training a model based on a data set with known labels, so that the model learns the mapping relationship between the input and the output. The model adjusts the parameters by comparing the difference between the predicted value and the true label, and finally realizes accurate prediction of new data. During the training process, the model parameters can be adjusted by using the minimum loss function to minimize the difference between the output label of the model and the input data. The loss function can be mean square error loss function, cross-entropy loss function, etc. The loss function is used to measure the difference between the predicted result of the model and the true result. The purpose is to minimize the loss function value by adjusting the model parameters, so as to improve the prediction accuracy.

[0057] The pre-trained task type classification model can be a task type classification model based on deep learning or machine learning, such as a Transformer model, CNN, etc.

[0058] The above-mentioned region segmentation can be understood as a process of dividing each work interface image in the work interface image sequence into multiple functional regions with different semantic meanings according to its functional attributes and visual boundaries.

[0059] The above-mentioned semantic recognition can be understood as a process of further analyzing the specific content in each functional region, so as to understand the actual function, meaning or user intention represented by each region in the current work context of the user.

[0060] The above-mentioned semantic label can be one or more identifiers that can accurately describe the function, content or user intention of the work interface image.

[0061] The user operation behavior can be a series of operation behaviors of the target user when using the software or application, such as keyboard operation, mouse operation, application and window focus activity, and the like. The keyboard operation includes high frequency, continuous tapping, long pause after short input, use of specific shortcut keys (such as Ctrl+S (save)), and the like, the mouse operation includes click frequency and mode, movement trajectory and speed, drag operation, and the like, and the application and window focus activity includes window switching frequency, application internal tab switching, and the like.

[0062] The monitoring can be understood as a process of tracking and recording the user operation behavior of each work interface image in the work interface image sequence.

[0063] The user operation log records the operation process and result of the target user in the system.

[0064] The current task type can be a kind of task that the target user is currently performing, such as document editing, data analysis, and the like, which is obtained according to the semantic label corresponding to the work interface image and the user operation log.

[0065] It should be noted that the trained task type classification model can be used to perform region segmentation and semantic recognition on each work interface image in the work interface image sequence, generate the corresponding semantic label, monitor the user operation behavior of each work interface image in the work interface image sequence, generate the user operation log, and obtain the current task type of the target user according to the semantic label and the user operation log, which can help the user to quickly identify the current task type and improve the work efficiency.

[0066] Optionally, in the step of calculating the interface change rate of the work interface image sequence, a first interface image and a second interface image of adjacent images can be extracted from the interface image sequence; a color feature deviation rate, a texture feature deviation rate and a semantic feature deviation rate between the first interface image and the second interface image are calculated; and the color feature deviation rate, the texture feature deviation rate and the semantic feature deviation rate are weighted and fused to obtain the interface change rate of the work interface image sequence.

[0067] In the embodiment of the application, the first interface image and the second interface image are adjacent interface images.

[0068] The color feature deviation rate can be understood as the difference degree between the first interface image and the second interface image in color distribution, hue and brightness.

[0069] The texture feature deviation rate can be understood as the difference degree between the first interface image and the second interface image in surface details, repetitive patterns, structural texture, and the like.

[0070] The semantic feature deviation rate can be understood as a difference degree between the first interface image and the second interface image in functional intention, core content and working context.

[0071] The weighted fusion processing can be understood as a processing procedure of assigning weights to the color feature deviation rate, the texture feature deviation rate and the semantic feature deviation rate, and then performing weighted average or combination to obtain the interface change rate of the working interface image sequence.

[0072] The interface change rate can be an interface change rate corresponding to the target user working interface image sequence obtained by performing the weighted fusion processing on the color feature deviation rate, the texture feature deviation rate and the semantic feature deviation rate. The interface change rate is used to measure the frequency, range or difference degree of user perception of interface update.

[0073] It should be noted that the first interface image and the second interface image of adjacent images can be extracted from the working interface image sequence, and the color feature deviation rate, the texture feature deviation rate and the semantic feature deviation rate between the first interface image and the second interface image are calculated. The color feature deviation rate, the texture feature deviation rate and the semantic feature deviation rate are weighted and fused to obtain the interface change rate of the working interface image sequence. The interface change rate can be used to measure the change degree between two images.

[0074] Optionally, in the step of calculating the color feature deviation rate and the texture feature deviation rate between the first interface image and the second interface image, the first image feature extraction processing can be performed on the first interface image to obtain the first image feature corresponding to the first interface image. The second image feature extraction processing is performed on the second interface image to obtain the second image feature corresponding to the second interface image. The difference degree between the first image feature and the second image feature in the color space is calculated to obtain the color feature deviation rate. The difference degree between the first image feature and the second image feature in the texture structure is calculated to obtain the texture feature deviation rate.

[0075] In the embodiment of the application, the first image feature extraction processing can be a processing procedure of extracting feature information capable of representing the image from the first interface image. The feature information can be color, texture, etc.

[0076] The first image feature can be an image feature corresponding to the first interface image.

[0077] The second image feature extraction processing can be a processing procedure of extracting feature information capable of representing the image from the second interface image.

[0078] The second image feature can be an image feature corresponding to the second interface image.

[0079] It can be understood that the first interface image and the second interface image are adjacent interface images.

[0080] It should be noted that the interface image can be analyzed by computer vision technology to extract feature information that can represent the content of the interface image. The above computer vision technology is a science that enables machines to "see" and understand visual information through image acquisition, processing and analysis. The core of computer vision technology is to simulate the human visual system to extract information from images or videos to complete tasks such as recognition, detection and measurement.

[0081] The color feature deviation rate can be the difference between the first image feature and the second image feature in color distribution, hue and brightness.

[0082] The color space is used to describe color, and the color space defines the representation of color by a set of numerical values, such as RGB, CMYK, etc.

[0083] Further, the difference between the first image feature and the second image feature in the color space is calculated to obtain the color feature deviation rate. Specifically, the color feature deviation rate between the first image feature and the second image feature can be obtained by calculating the color histogram difference between the first image feature and the second image feature. If the color feature deviation rate between the first image feature and the second image feature is small, it indicates that the difference between the first interface image and the second interface image is small; if the color feature deviation rate between the first image feature and the second image feature is large, it indicates that the color difference between the interface image and the second interface image is large. The color histogram is a tool that reflects the overall color distribution of the image by counting the frequency of each color pixel in the image. The color histogram reflects the color tone composition of the image by counting the frequency of different color values.

[0084] The texture structure can be a visual feature of the surface of an object in the image.

[0085] The texture feature deviation rate can be the difference between the first image feature and the second image feature in surface details, repetitive patterns, structural texture, etc.

[0086] The texture feature deviation rate can be obtained by calculating the difference between the first image feature and the second image feature in the texture structure. Specifically, the texture feature of the image can be extracted using a gray level co-occurrence matrix (GLCM), and then the GLCM difference between the first image feature and the second image feature is calculated. The gray level co-occurrence matrix (GLCM) is a statistical method for describing texture features by analyzing the spatial correlation of adjacent pixel gray values in an image. The gray level co-occurrence matrix (GLCM) reflects the texture structure of the image by calculating the gray level combination frequency of pixel pairs at a certain distance and direction.

[0087] Optionally, in the step of calculating the semantic feature deviation rate between the first interface image and the second interface image, the first semantic recognition can be performed on the first interface image to obtain a first semantic feature vector corresponding to the first interface image; the second semantic recognition can be performed on the second interface image to obtain a second semantic feature vector corresponding to the second interface image; and the feature distance between the first semantic feature vector and the second semantic feature vector is calculated to obtain the semantic feature deviation rate.

[0088] In the embodiment of the present application, the first semantic recognition described above can be a processing process of analyzing and understanding the first interface image and extracting semantic information of objects, scenes, etc. in the image.

[0089] The first semantic feature vector described above can be a semantic feature vector corresponding to the first interface image, and the first semantic feature vector includes semantic features of objects, scenes, etc. in the first interface image.

[0090] The second semantic recognition described above can be a processing process of analyzing and understanding the second interface image and extracting semantic information of objects, scenes, etc. in the image.

[0091] The second semantic feature vector described above can be a semantic feature vector corresponding to the second interface image, and the second semantic feature vector includes semantic features of objects, scenes, etc. in the second interface image.

[0092] The feature distance described above can be understood as the distance of the first semantic feature vector and the second semantic feature vector in space. The Euclidean distance or the like can be used to measure the distance of the first semantic feature vector and the second semantic feature vector in space. The Euclidean distance is the real straight line distance between two points in Euclidean space, and the smaller the Euclidean distance, the more similar the two are, and vice versa, the larger the Euclidean distance, the more dissimilar the two are.

[0093] The semantic feature deviation rate described above can be the difference degree between the first semantic feature and the second semantic feature in functional intent, core content and working context.

[0094] It should be noted that the first interface image and the second interface image are adjacent interface images. The semantic recognition processing of the interface image can be performed by computer vision technology to obtain the semantic feature vector corresponding to the interface image, and the computer vision technology is a science that enables machines to "see" and understand visual information through image acquisition, processing and analysis. The core of computer vision technology is to simulate the human visual system to extract information from images or videos to complete recognition, detection, measurement and other tasks.

[0095] Optionally, in the step of collecting load data of the cloud computer system in real time and predicting load states of each processing channel in a preset time period, historical and real-time load data of other users in the cloud computer system in the same collaborative session as the target user can be collected in real time; the load data is input into a preset load prediction model to predict load states of each processing channel in the next working stage of the collaborative session in the preset time period.

[0096] In the embodiments of the present application, the above-mentioned preset load prediction model is obtained by training a pre-trained load prediction model based on a load training data set, and the load training data set includes historical load data and corresponding historical load state label data, real-time load data and corresponding real-time load state label data. The above-mentioned pre-trained load prediction model can be a load prediction model based on deep learning or constructed by deep learning, such as LSTM neural network, support vector machine (SVM) and the like. The above-mentioned training can be supervised training, which is a process of training a model by a data set with known labels to make the model learn the mapping relationship between input and output. The model continuously adjusts the parameters by comparing the difference between the predicted value and the real label, and finally realizes accurate prediction of new data. In the training process, the model can be adjusted in parameters by a minimum loss function to minimize the difference between the label of the output of the model and the input data. The above-mentioned loss function can be mean square error loss function, cross-entropy loss function and the like. The loss function is used to measure the difference between the predicted result of the model and the real result, and the purpose is to minimize the loss function value by adjusting the parameters of the model, so as to improve the prediction accuracy.

[0097] The above-mentioned collaborative session can be a process in which multiple users participate and interact at the same time.

[0098] The above-mentioned historical load data can be the load condition of the cloud computer system in the past period of time, such as CPU usage rate and the like.

[0099] The above-mentioned real-time load data can be the load condition generated in real time.

[0100] The above-mentioned other users can be users in the same collaborative session as the target user.

[0101] The above-mentioned preset time period can be a future period of time set in advance, such as 5 seconds, 8 seconds, 10 seconds and the like.

[0102] The load state of each processing channel can be the load state of each processing channel in the preset time period of the next working stage, such as the usage of CPU, memory and the like It should be noted that the historical and real-time load data of other users in the cloud computer system in the same collaborative session as the target user can be collected in real time, and the load data can be input into a preset load prediction model to predict the load state of each processing channel in the next working stage of the collaborative session within a preset time period, so that resources can be more reasonably allocated and utilized, and work efficiency can be improved.

[0103] Optionally, in the step of determining the target processing channel in the candidate processing channel set based on the load state, a corresponding comprehensive optimization score can be calculated for each candidate processing channel in the candidate processing channel set based on a multi-objective optimization algorithm; and the candidate processing channel with the highest comprehensive optimization score in the candidate processing channel set can be selected as the target processing channel according to the load state.

[0104] In the embodiments of the present application, the multi-objective optimization algorithm can be a mathematical method for solving optimization problems containing multiple conflicting objective functions simultaneously, and the core goal of the multi-objective optimization algorithm is to find a set of “Pareto optimal solutions”, that is, in these solutions, no solution can be better than other solutions in all objectives. The multi-objective optimization algorithm can be a genetic algorithm (GA), a particle swarm optimization, etc.

[0105] The candidate processing channel set can be a plurality of candidate processing channels most suitable for processing the current task and the interface change rate selected from a predefined processing channel resource pool according to the current task type and the interface change rate.

[0106] The candidate processing channel set can be a priority queue set for classifying and scheduling different service flows.

[0107] The comprehensive optimization score is used to measure the pros and cons of the candidate processing channel in multiple aspects of performance.

[0108] Specifically, the corresponding comprehensive optimization score can be calculated for each candidate processing channel in the candidate processing channel set according to the multi-objective optimization algorithm, and the candidate processing channel with the highest comprehensive optimization score in the candidate processing channel set can be selected as the target processing channel according to the load state, so that the overall performance of the system can be improved.

[0109] It should be noted that the optimization objectives of the multi-objective optimization algorithm can include minimizing the overall system response delay and maximizing the resource utilization rate, and the input parameters of the multi-objective optimization algorithm at least include the predicted load state of each processing channel, the inherent matching degree of each channel to the current task type, and the processing efficiency coefficient of each channel to the interface change rate. When the candidate processing channel with the highest comprehensive optimization score is occupied, the target processing channel can be selected from the candidate channels ranked only lower than the highest candidate channel.

[0110] As shown in Figure 2 An embodiment of the present application provides a cloud computer-based collaborative work device, which comprises: A generation module 201 is configured to collect work interface image of any target user in a plurality of clients and generate a work interface image sequence; A first determination module 202 is configured to perform semantic feature extraction on the work interface image sequence to determine a current task type corresponding to the target user; A calculation module 203 is configured to calculate an interface change rate of the work interface image sequence; A selection module 204 is configured to select a candidate processing channel set from a predefined processing channel resource pool according to the current task type and the interface change rate; A prediction module 205 is configured to collect load data of a cloud computer system in real time and predict load states of each processing channel in a preset time period; A second determination module 206 is configured to determine a target processing channel from the candidate channel set based on the load state; A collaborative processing module 207 is configured to perform cloud data processing on the work interface image sequence of the target user according to the target processing channel, generate a cloud processing result, and perform collaborative processing on the work of the target user according to the cloud processing result.

[0111] Optionally, the first determination module 202 is further configured to perform region segmentation and semantic recognition on each work interface image in the work interface image sequence by using a trained task type classification model to generate corresponding semantic labels, monitor user operation behaviors of each work interface image in the work interface image sequence to generate user operation logs, and obtain the current task type corresponding to the target user based on the semantic labels and the user operation logs. The trained task type classification model is obtained by training a pre-trained task type classification model by using a training data set, the training data set includes sample work interface image data, semantic annotation data and user operation annotation data of the sample work interface image data, and the pre-trained task type classification model outputs a task type of the sample work interface image data.

[0112] Optionally, the computing module 203 is further configured to extract a first interface image and a second interface image of adjacent images from the working interface image sequence; calculate a color feature deviation rate, a texture feature deviation rate and a semantic feature deviation rate between the first interface image and the second interface image; and perform weighted fusion processing on the color feature deviation rate, the texture feature deviation rate and the semantic feature deviation rate to obtain an interface change rate of the working interface image sequence.

[0113] Optionally, the computing module 203 is further configured to perform first image feature extraction processing on the first interface image to obtain a first image feature corresponding to the first interface image; perform second image feature extraction processing on the second interface image to obtain a second image feature corresponding to the second interface image; calculate a difference degree between the first image feature and the second image feature in a color space to obtain a color feature deviation rate; and calculate a difference degree between the first image feature and the second image feature in a texture structure to obtain a texture feature deviation rate.

[0114] Optionally, the computing module 203 is further configured to perform first semantic recognition on the first interface image to obtain a first semantic feature vector corresponding to the first interface image; perform second semantic recognition on the second interface image to obtain a second semantic feature vector corresponding to the second interface image; and calculate a feature distance between the first semantic feature vector and the second semantic feature vector to obtain a semantic feature deviation rate.

[0115] Optionally, the prediction module 205 is further configured to collect historical and real-time load data of other users in a cloud computer system in a same collaborative session as the target user in real time; input the load data into a preset load prediction model to predict load states of each processing channel in a preset time period in a next working stage of the collaborative session, the preset load prediction model being obtained by training a pre-trained load prediction model by using a load training data set, and the load training data set including historical load data and corresponding historical load state label data, real-time load data and corresponding real-time load state label data.

[0116] Optionally, the second determination module 206 is further configured to calculate a corresponding comprehensive optimization score for each candidate processing channel in the candidate processing channel set based on a multi-objective optimization algorithm; and select a candidate processing channel with the highest comprehensive optimization score in the candidate processing channel set as the target processing channel according to the load state.

[0117] As shown in Figure 3 The embodiment of the present application also provides an electronic device, which comprises a processor, and the processor can execute any one of the above-mentioned collaborative working methods based on a cloud computer.

[0118] In particular, the computer program for implementing the cloud computer-based collaboration method comprises a processor 301 and a memory 302, and is stored in the memory 302 and executable on the processor 301. The processor 301 executes the computer program for implementing the cloud computer-based collaboration method stored in the memory 302, and performs the following steps: Collecting a work interface image of any target user in a plurality of clients, and generating a work interface image sequence; Extracting semantic features of the work interface image sequence to determine the current task type corresponding to the target user; Calculating the interface change rate of the work interface image sequence; According to the current task type and the interface change rate, a candidate processing channel set is selected from a predefined processing channel resource pool; Real-time collection of load data of the cloud computer system, and prediction of the load state of each processing channel in a preset time period; Based on the load state, a target processing channel is determined from the candidate channel set; According to the target processing channel, the work interface image sequence of the target user is processed by cloud data, a cloud processing result is generated, and the work of the target user is collaboratively processed according to the cloud processing result.

[0119] Optionally, the processor 301 performs the semantic feature extraction of the work interface image sequence to determine the current task type corresponding to the target user, comprising: Segmenting and identifying the semantic labels of each work interface image in the work interface image sequence through a trained task type classification model; Monitoring the user operation behavior of each work interface image in the work interface image sequence to generate a user operation log; Based on the semantic labels and the user operation log, the current task type corresponding to the target user is obtained, the trained task type classification model is obtained by training a pre-trained task type classification model using a training data set, the training data set includes sample work interface image data, semantic annotation data and user operation annotation data of the sample work interface image data, and the pre-trained task type classification model outputs the task type of the sample work interface image data.

[0120] Optionally, the processor 301 performs the calculation of the interface change rate of the work interface image sequence, comprising: Extracting a first interface image and a second interface image of adjacent images from the work interface image sequence; calculate a color feature deviation rate, a texture feature deviation rate and a semantic feature deviation rate between the first interface image and the second interface image; perform weighted fusion processing on the color feature deviation rate, the texture feature deviation rate and the semantic feature deviation rate to obtain an interface change rate of the working interface image sequence.

[0121] Optionally, the calculation of the color feature deviation rate and the texture feature deviation rate between the first interface image and the second interface image performed by the processor 301 comprises: perform first image feature extraction processing on the first interface image to obtain a first image feature corresponding to the first interface image; perform second image feature extraction processing on the second interface image to obtain a second image feature corresponding to the second interface image; calculate a difference degree between the first image feature and the second image feature in a color space to obtain a color feature deviation rate; calculate a difference degree between the first image feature and the second image feature in a texture structure to obtain a texture feature deviation rate.

[0122] Optionally, the calculation of the semantic feature deviation rate between the first interface image and the second interface image performed by the processor 301 comprises: perform first semantic recognition on the first interface image to obtain a first semantic feature vector corresponding to the first interface image; perform second semantic recognition on the second interface image to obtain a second semantic feature vector corresponding to the second interface image; calculate a feature distance between the first semantic feature vector and the second semantic feature vector to obtain a semantic feature deviation rate.

[0123] Optionally, the real-time collection of load data of the cloud computer system and the prediction of load states of each processing channel in a preset time period performed by the processor 301 comprises: real-time collection of historical and real-time load data of other users in the cloud computer system who are in the same collaborative session as the target user; input the load data into a preset load prediction model to predict load states of each processing channel in a preset time period in a next working stage of the collaborative session, the preset load prediction model being obtained by training a pre-trained load prediction model by using a load training data set, the load training data set comprising historical load data and corresponding historical load state label data, real-time load data and corresponding real-time load state label data.

[0124] Optionally, the processor 301 performs the determining the target processing channel from the candidate processing channel set based on the load state, comprising: calculating a corresponding comprehensive optimization score for each candidate processing channel in the candidate processing channel set based on a multi-objective optimization algorithm; selecting a candidate processing channel with the highest comprehensive optimization score from the candidate processing channel set as the target processing channel according to the load state.

[0125] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, the computer program is executed by a processor to realize each process of the cloud computer based collaborative work method provided by the embodiment of the present application, and the same technical effects can be achieved, and details are not repeated here.

[0126] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiments of the method can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM).

[0127] The above only describes the preferred embodiments of the present application, and of course cannot limit the scope of the present application, so equivalent changes made according to the claims of the present application are still within the scope of the present application.

Claims

1. A cloud computer-based collaborative work method, characterized by, The method comprises: collecting a work interface image of any target user in a plurality of clients, and generating a work interface image sequence; performing semantic feature extraction on the work interface image sequence to determine a current task type corresponding to the target user; calculating an interface change ratio of the work interface image sequence; selecting a candidate processing channel set from a predefined processing channel resource pool according to the current task type and the interface change ratio; collecting load data of a cloud computer system in real time, and predicting load states of each processing channel in a preset time period; determining a target processing channel from the candidate channel set based on the load states; performing cloud data processing on the work interface image sequence of the target user according to the target processing channel, generating a cloud processing result, and performing collaborative processing on the work of the target user according to the cloud processing result.

2. The method of claim 1, wherein, The semantic feature extraction on the work interface image sequence to determine the current task type corresponding to the target user comprises: performing region segmentation and semantic recognition on each work interface image in the work interface image sequence through a trained task type classification model to generate corresponding semantic labels; monitoring user operation behaviors of each work interface image in the work interface image sequence to generate user operation logs; obtaining the current task type corresponding to the target user based on the semantic labels and the user operation logs, wherein the trained task type classification model is obtained by training a pre-trained task type classification model through a training data set, the training data set comprises sample work interface image data, semantic annotation data and user operation annotation data of the sample work interface image data, and the pre-trained task type classification model outputs a task type of the sample work interface image data.

3. The method of claim 1, wherein, The calculation of the interface change ratio of the work interface image sequence comprises: extracting a first interface image and a second interface image of adjacent images from the work interface image sequence; calculating color feature deviation rate, texture feature deviation rate and semantic feature deviation rate between the first interface image and the second interface image; performing weighted fusion processing on the color feature deviation rate, the texture feature deviation rate and the semantic feature deviation rate to obtain the interface change ratio of the work interface image sequence.

4. The method of claim 3, wherein, The calculation of the color feature deviation rate and the texture feature deviation rate between the first interface image and the second interface image comprises: performing first image feature extraction processing on the first interface image to obtain first image features corresponding to the first interface image; performing second image feature extraction processing on the second interface image to obtain second image features corresponding to the second interface image; calculating a difference degree between the first image features and the second image features in a color space to obtain a color feature deviation rate; calculating a difference degree between the first image features and the second image features in a texture structure to obtain a texture feature deviation rate.

5. The method of claim 3, wherein, The calculation of the semantic feature deviation rate between the first interface image and the second interface image comprises: performing first semantic recognition on the first interface image to obtain a first semantic feature vector corresponding to the first interface image; performing second semantic recognition on the second interface image to obtain a second semantic feature vector corresponding to the second interface image; calculating a feature distance between the first semantic feature vector and the second semantic feature vector to obtain a semantic feature deviation rate.

6. The method of claim 1, wherein, The load data of the cloud computer system is collected in real time, and the load states of each processing channel in a preset time period are predicted, including: collecting historical and real-time load data of other users in the same collaborative session as the target user in the cloud computer system in real time; inputting the load data into a preset load prediction model to predict the load states of each processing channel in a preset time period in the next working stage of the collaborative session, the preset load prediction model being obtained by training a pre-trained load prediction model based on a load training data set, and the load training data set including historical load data and corresponding historical load state label data, real-time load data and corresponding real-time load state label data.

7. The method of claim 1, wherein, The target processing channel is determined from the candidate processing channel set based on the load state, including: calculating a corresponding comprehensive optimization score for each candidate processing channel in the candidate processing channel set based on a multi-objective optimization algorithm; selecting the candidate processing channel with the highest comprehensive optimization score from the candidate processing channel set as the target processing channel according to the load state. 8.A cloud computer-based collaborative work device, characterized by comprising: The collaborative working device based on the cloud computer includes: a generation module configured to collect working interface image of any target user in a plurality of clients and generate a working interface image sequence; a first determination module configured to perform semantic feature extraction on the working interface image sequence to determine a current task type corresponding to the target user; a calculation module configured to calculate an interface change rate of the working interface image sequence; a selection module configured to select a candidate processing channel set from a predefined processing channel resource pool according to the current task type and the interface change rate; a prediction module configured to collect load data of the cloud computer system in real time and predict load states of each processing channel in a preset time period; a second determination module configured to determine a target processing channel from the candidate processing channel set based on the load state; a collaborative processing module configured to perform cloud data processing on the working interface image sequence of the target user according to the target processing channel, generate a cloud processing result, and perform collaborative processing on the work of the target user according to the cloud processing result.

9. An electronic device, comprising: including: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the collaborative working method based on the cloud computer according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the collaborative working method based on the cloud computer according to any one of claims 1 to 7.

Images