Graphical interface preview method and device based on MJPEG protocol

Abstract

The invention relates to the technical field of video coding and transmission, and provides a graphical interface previewing method and device based on an MJPEG protocol, and the method comprises the steps: capturing a graphical interface of a cloud mobile phone screen in real time based on the MJPEG protocol, generating a screen graphical interface stream and a corresponding MUPEG image stream feature set, the feature number is the same as the frame number, and the MUPEG image stream feature set comprises a feature graph and a parameter set; carrying out compressed encoding on the feature set to obtain an encoded feature set; performing video coding on the screen stream according to the coding feature set to generate a video bit stream; if the capacity of the coding feature set is smaller than or equal to the threshold value, embedding the coding feature set into the video bit stream to form an embedded bit stream, and performing embedded transmission; if the capacity is greater than the threshold, the video bitstream and the coding feature set are transmitted, respectively. According to the method, high-efficiency and low-delay screen picture transmission is realized through feature compression and differential transmission strategies. The data transmission amount can be reduced, and the real-time preview efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video encoding and transmission. More specifically, the present invention relates to a method and apparatus for previewing a graphical interface based on the MJPEG protocol. Background Art

[0002] In today's digital age, with the rapid development of cloud technology, the cloud real device technology has gradually become an important application form, which allows users to remotely access and operate virtual mobile phone devices through the network to perform various function tests, application demonstrations, etc. However, when previewing the graphical interface of the mobile phone screen in the cloud real device, some technical challenges are faced. Traditional graphical interface preview methods usually adopt a simple image transmission method, that is, each frame of the mobile phone screen image is directly encoded and transmitted to the client for display. Although this method is relatively simple to implement, it will occupy a large amount of bandwidth resources during the transmission process, resulting in a high transmission delay. Especially in an unstable network environment, it is easy to appear stuck, seriously affecting the user experience. In addition, when encoding images, traditional methods often do not fully consider the correlation between image frames, resulting in low encoding efficiency and unable to effectively reduce the data volume, further exacerbating the transmission pressure.

[0003] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the prior art: it is impossible to efficiently perform real-time preview of the graphical interface of the mobile phone screen in the cloud real device, and there is a large bandwidth occupation and high latency during the data transmission process, making it difficult to meet the user's demand for smooth preview. Summary of the Invention

[0004] The present invention provides a method, apparatus, device and medium for previewing a graphical interface based on the MJPEG protocol.

[0005] In the first aspect of the present invention, a method for previewing a graphical interface based on the MJPEG protocol is provided, including: based on the MJPEG protocol, capturing the graphical interface of the mobile phone screen in the cloud real machine in real time to obtain a screen graphical interface stream and determining a corresponding MJPEG image stream feature set, wherein the number of image stream features in the MJPEG image stream feature set is the same as the number of frames of the screen graphical interface stream, and the image stream features include image stream feature maps and corresponding feature parameter sets; performing compression encoding on each image stream feature in the MJPEG image stream feature set to obtain an encoded image stream feature set; performing video encoding on the screen graphical interface stream according to the encoded image stream feature set to obtain a video bitstream; in response to the capacity of the encoded image stream feature set being less than or equal to a preset threshold, adding the encoded image stream feature set to the video bitstream to obtain an embedded image stream bitstream; performing embedded transmission on the embedded image stream bitstream, wherein the embedded image stream transmission identifier in the embedded image stream bitstream is 1; in response to the capacity of the encoded image stream feature set being greater than the preset threshold, transmitting the video bitstream and the encoded image stream feature set respectively, wherein the embedded image stream transmission identifier in the video bitstream is 0.

[0006] Further, the performing compression encoding on each image stream feature in the MJPEG image stream feature set to obtain an encoded image stream feature set includes: obtaining an updated image stream identifier list, wherein the number of updated image stream identifiers in the updated image stream identifier list is the same as the number of frames of the screen graphical interface stream; independently encoding the first corresponding image stream feature in the MJPEG image stream feature set to obtain a first encoded image stream feature and setting the corresponding updated image stream identifier to 1; sequentially performing the following encoding steps on each image stream feature in the MJPEG image stream feature set except the first one to obtain an initial encoded image stream feature set: in response to the current image stream feature being different from the previous image stream feature, performing differential encoding on the current image stream feature to obtain an initial encoded image stream feature and setting the corresponding updated image stream identifier to 1; in response to the current image stream feature being the same as the previous image stream feature, setting the corresponding updated image stream identifier to 0; determining the first encoded image stream feature and the initial encoded image stream feature set as the encoded image stream feature set.

[0007] Further, independently encoding the first image stream feature corresponding to the MJPEG image stream feature set to obtain a first encoded image stream feature includes: determining the first image stream feature corresponding to the MJPEG image stream feature set as a target image stream feature; optimizing feature extraction of the target image stream feature to obtain an optimized image stream feature; compressing the optimized image stream feature to obtain a compressed image stream feature; and performing parameter quantization on the compressed feature parameter set corresponding to the compressed image stream feature to obtain the first encoded image stream feature.

[0008] Further, optimizing feature extraction of the target image stream feature to obtain an optimized image stream feature includes: adding an intermediate fully-connected layer that meets a preset condition before the fully-connected layer in the target image stream feature to obtain an initial image stream feature, where the fully-connected layer corresponds to an input feature and an output feature, the intermediate fully-connected layer is used to obtain an intermediate output feature according to the input feature, the fully-connected layer is used to obtain an updated output feature according to the intermediate output feature, the dimension of the intermediate output feature is smaller than the dimensions of the input feature and the output feature, and the dimension of the updated output feature is equal to the dimension of the output feature; and replacing the convolutional layer in the initial image stream feature with depthwise separable convolution to obtain the optimized image stream feature.

[0009] Further, performing parameter quantization on the compressed feature parameter set corresponding to the compressed image stream feature to obtain the first encoded image stream feature includes: determining the difference between the largest and smallest compressed feature parameters corresponding to the compressed feature parameter set of the compressed image stream feature as a dynamic range value; determining the ratio of the dynamic range value to a preset maximum representation value as a scaling factor; determining the ratio of the smallest compressed feature parameter to the scaling factor as an offset; performing scaling mapping on each compressed feature parameter in the compressed feature parameter set according to the scaling factor and the offset to generate a mapped feature parameter to obtain a set of mapped feature parameters; and determining the first encoded image stream feature corresponding to the compressed image stream feature according to the set of mapped feature parameters.

[0010] Further, performing differential encoding on the current image stream feature to obtain an initial encoded image stream feature includes: determining a difference feature map between the current image stream feature and the corresponding previous image stream feature; determining the feature parameter difference value between the current image stream feature and the corresponding previous image stream feature as a difference feature parameter value to obtain a set of difference feature parameter values; and determining the difference feature map and the set of difference feature parameter values as the initial encoded image stream feature corresponding to the current image stream feature.

[0011] In a second aspect of the present invention, there is provided a graphical interface preview device based on the MJPEG protocol, including: a first video encoding unit configured to, based on the MJPEG protocol, perform real-time capture on the graphical interface of the mobile phone screen in the cloud real machine to obtain a screen graphical interface stream and determine a corresponding MJPEG image stream feature set, wherein the number of image stream features in the MJPEG image stream feature set is the same as the number of frames of the screen graphical interface stream, and the image stream feature includes an image stream feature map and a corresponding feature parameter set; a compression encoding unit configured to perform compression encoding on each image stream feature in the MJPEG image stream feature set to obtain an encoded image stream feature set.

[0012] A second video encoding unit configured to perform video encoding on the screen graphical interface stream according to the encoded image stream feature set to obtain a video bitstream; an adding unit configured to, in response to the capacity of the encoded image stream feature set being less than or equal to a preset threshold, add the encoded image stream feature set to the video bitstream to obtain an embedded image stream bitstream; an embedded transmission unit configured to perform embedded transmission on the embedded image stream bitstream, wherein the embedded image stream transmission identifier in the embedded image stream bitstream is 1; a separate transmission unit configured to, in response to the capacity of the encoded image stream feature set being greater than the preset threshold, perform separate transmission on the video bitstream and the encoded image stream feature set, wherein the embedded image stream transmission identifier in the video bitstream is 0.

[0013] Further, the compression encoding unit includes: an independent encoding subunit configured to perform independent encoding on the first corresponding image stream feature in the MJPEG image stream feature set to obtain a first encoded image stream feature; a differential encoding subunit configured to perform differential encoding on each image stream feature in the MJPEG image stream feature set except the first one to obtain an initial encoded image stream feature set; a feature identifier setting subunit configured to set a corresponding updated image stream identifier.

[0014] In a third aspect of the present invention, there is provided an electronic device, the electronic device including: at least one processor, a memory, and an input / output unit; wherein, the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the method described in any one of the first aspects.

[0015] In a fourth aspect of the present invention, there is provided a computer-readable storage medium, which includes instructions that, when running on a computer, cause the computer to execute the method described in any one of the first aspects.

[0016] The above embodiments of the present invention have at least the following beneficial effects: The present invention can improve the efficiency and quality of previewing the graphical user interface of the mobile phone screen in the cloud real machine. By capturing the screen graphical user interface stream in real time based on the MJPEG protocol and determining the corresponding image stream feature set, and then performing a series of operations such as compression encoding on the image stream features, it is possible to effectively reduce the data transmission volume while ensuring the integrity of the image information. When the capacity of the encoded image stream feature set is less than or equal to the preset threshold, it is added to the video bitstream for embedded transmission. This flexible transmission method further optimizes the data transmission process, improves the stability and reliability of the transmission, reduces the transmission delay, and enables users to preview the graphical user interface more smoothly.

[0017] In addition, the present invention can also enhance the adaptability and flexibility of the system. The combination of multiple encoding methods such as independent encoding and differential encoding of the image stream features can be flexibly processed according to the differences between different frames, further improving the encoding efficiency. At the same time, through operations such as feature extraction optimization and parameter quantization, key information can be better retained and the image quality can be improved. This fine processing of the image stream features and the selection of multiple transmission methods enable the solution to better adapt to different network environments and application scenarios, meet the preview requirements of users under different conditions, and provide strong support for the development of cloud real machine technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present invention will become readily understood. In the drawings, several embodiments of the present invention are shown by way of illustration and not limitation, wherein: Figure 1 It is a schematic flowchart of a graphical user interface preview method based on the MJPEG protocol provided by an embodiment of the present invention.

[0019] Figure 2 It is a schematic structural diagram of a graphical user interface preview device based on the MJPEG protocol provided by an embodiment of the present invention.

[0020] Figure 3 It schematically shows a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and then implement the present invention, and do not limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to be able to fully convey the scope of the present invention to those skilled in the art.

[0022] Those skilled in the art know that the embodiments of the present invention can be implemented as a device, apparatus, equipment, method, or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0023] It should be noted that the number of any elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0024] The following refers to Figure 1 , Figure 1 which is a schematic flowchart of a graphical interface preview method based on the MJPEG protocol provided for an embodiment of the present invention. As Figure 1 shown, a graphical interface preview method based on the MJPEG protocol includes: S1. Based on the MJPEG protocol, perform real-time capture on the mobile phone screen graphical interface in the cloud real machine to obtain a screen graphical interface stream and determine the corresponding MJPEG image stream feature set. Among them, the number of image stream features in the MJPEG image stream feature set is the same as the number of frames of the screen graphical interface stream, and the image stream feature includes an image stream feature map and a corresponding feature parameter set.

[0025] S2. Perform compression encoding on each image stream feature in the MJPEG image stream feature set to obtain an encoded image stream feature set.

[0026] S3. According to the encoded image stream feature set, perform video encoding on the screen graphical interface stream to obtain a video bitstream.

[0027] S4. In response to the capacity of the encoded image stream feature set being less than or equal to a preset threshold, add the encoded image stream feature set to the video bitstream to obtain an embedded image stream bitstream.

[0028] S5. Perform embedded transmission on the embedded image stream bitstream, where the embedded image stream transmission identifier in the embedded image stream bitstream is 1.

[0029] S6. In response to the capacity of the encoded image stream feature set being greater than the preset threshold, perform transmission on the video bitstream and the encoded image stream feature set respectively, where the embedded image stream transmission identifier in the video bitstream is 0.

[0030] It should be noted that the present invention proposes a graphical interface preview method based on the MJPEG protocol. The core lies in efficiently processing and transmitting the graphical interface of the mobile phone screen captured in real time in the cloud real machine to achieve a smooth preview effect. Among them, the cloud real machine refers to a virtual mobile phone environment built based on cloud computing technology, and users can remotely access and operate these virtual devices through the network. The MJPEG protocol is an image compression protocol based on Motion JPEG, which is used to independently encode each frame of the video stream and has the characteristics of simplicity and efficiency. By determining the MJPEG image stream feature set corresponding to the screen graphical interface stream, the present invention can effectively extract the key information of each frame of the image, and then achieve the efficient compression and transmission of the image stream.

[0031] Specifically, the screen graphical interface stream mentioned in the present invention refers to a series of consecutive image frames generated during the operation of the mobile phone screen in the cloud real machine, and these image frames reflect the real-time display content of the mobile phone screen. The MJPEG image stream feature set refers to a set of feature information extracted from the screen graphical interface stream. Each feature information corresponds to a frame of the image and includes an image stream feature map and a corresponding feature parameter set. The image stream feature map is an abstract representation of the visual content of the image frame, while the feature parameter set contains information such as quantization parameters related to the image frame. During the implementation process, by analyzing each frame of the image, its key visual features and related parameters can be extracted, so as to construct a complete MJPEG image stream feature set. For example, the feature parameter set can include parameters such as the brightness, contrast, and color distribution of the image, and these parameters can help the subsequent compression and encoding process better retain the key information of the image.

[0032] Preferably, when compressing and encoding each image stream feature in the MJPEG image stream feature set, various optimization strategies can be adopted. For example, for the first image stream feature, independent encoding can be performed, which means encoding it as an independent unit without relying on the information of other image frames. For subsequent image stream features, differential encoding can be used to encode only the different parts from the previous frame of the image stream feature, thus greatly reducing the amount of data after encoding. In addition, during the feature extraction optimization process, an intermediate fully connected layer can be added before the fully connected layer of the image stream feature. By adjusting the parameters of the intermediate fully connected layer, the dimension of the intermediate output feature is made smaller than that of the input feature and the output feature, thereby achieving feature dimensionality reduction and optimization. This optimization method can effectively reduce the computational amount and storage requirements while retaining the key feature information of the image. During the parameter quantization process, the dynamic range value between the maximum value and the minimum value in the compressed feature parameter set can be calculated, and the scaling factor and offset can be determined according to this dynamic range value, and then the feature parameters are scaled and mapped to generate a mapped feature parameter set, and finally the first encoded image stream feature is obtained.

[0033] In some embodiments, compressing and encoding each image stream feature in the MJPEG image stream feature set to obtain an encoded image stream feature set includes: obtaining an updated image stream identifier list, where the number of updated image stream identifiers in the updated image stream identifier list is the same as the number of frames of the screen graphical interface stream; independently encoding the first image stream feature in the MJPEG image stream feature set to obtain a first encoded image stream feature, and setting the corresponding updated image stream identifier to 1; sequentially performing the following encoding steps on each image stream feature in the MJPEG image stream feature set except the first one to obtain an initial encoded image stream feature set: in response to the current image stream feature being different from the previous image stream feature, performing differential encoding on the current image stream feature to obtain an initial encoded image stream feature, and setting the corresponding updated image stream identifier to 1; in response to the current image stream feature being the same as the previous image stream feature, setting the corresponding updated image stream identifier to 0; determining the first encoded image stream feature and the initial encoded image stream feature set as the encoded image stream feature set.

[0034] It should be noted that the present invention further refines the process of compressing and encoding the MJPEG image stream feature set. In this process, first, an updated image stream identifier list needs to be obtained, which is used to mark whether each frame of the image stream feature has been updated. The number of identifiers in the updated image stream identifier list is the same as the number of frames of the screen graphical interface stream, ensuring that each frame of the image has a corresponding identifier. For the first image stream feature in the MJPEG image stream feature set, it is processed by independent encoding, that is, without relying on the information of other frames, directly encoding it, and setting the corresponding updated image stream identifier to 1, indicating that this frame of the image stream feature is independently updated. For subsequent image stream features, by judging whether the current image stream feature is the same as the previous image stream feature, it is decided whether to perform differential encoding or directly skip encoding. If the current image stream feature is different from the previous image stream feature, perform differential encoding on the current image stream feature and set the corresponding updated image stream identifier to 1; if they are the same, set the corresponding updated image stream identifier to 0. Finally, combine the first encoded image stream feature obtained by independent encoding and the initial encoded image stream feature set obtained by differential encoding to form a complete encoded image stream feature set. This method can effectively reduce the amount of encoded data and improve the encoding efficiency.

[0035] Specifically, the updated image stream identifier list is a list used to track the update status of each frame of image stream features. The updated image stream identifier therein is a binary value used to indicate whether the corresponding image stream features have changed. For example, when the updated image stream identifier is 1, it means that the image stream features of this frame are different from the previous frame and need to be encoded for update; while when the updated image stream identifier is 0, it means that the image stream features of this frame are the same as the previous frame and there is no need to repeat the encoding. During the independent encoding process, the first image stream feature is determined as the target image stream feature, and a series of operations such as feature extraction optimization, feature compression, and parameter quantization are performed on it to generate the first encoded image stream feature. Differential encoding is for subsequent image stream features. By calculating the difference between the current image stream feature and the previous image stream feature, only the different parts are encoded, thus reducing the amount of encoded data. For example, if the current image stream feature and the previous image stream feature do not change in some regions or parameters, these parts of the encoding can be skipped, and only the changed parts are encoded. This differential encoding method can make full use of the correlation between image frames and further improve the encoding efficiency.

[0036] Preferably, when compressing and encoding the MJPEG image stream feature set, the processes of independent encoding and differential encoding can be further optimized. For independent encoding, a deep learning model can be used to optimize the feature extraction of the target image stream feature. For example, a convolutional neural network (CNN) model can be used. Input the target image stream feature map, extract the key features of the image through operations such as convolutional layers and pooling layers, and then perform compression and quantization processing on the extracted features to generate the first encoded image stream feature. During the differential encoding process, a more detailed analysis can be performed on the difference between the current image stream feature and the previous image stream feature. For example, the pixel-level difference between two frames of image stream features can be calculated to generate a difference feature map, and each pixel value in the difference feature map is quantized to obtain a set of difference feature parameter values. Then, the difference feature map and the set of difference feature parameter values are combined as the initial encoded image stream feature corresponding to the current image stream feature. In addition, according to actual application requirements, further optimization processing can be performed on the encoded image stream features. For example, through an adaptive coding rate control algorithm, the coding parameters are dynamically adjusted according to the network bandwidth and image quality requirements to achieve the best coding effect.

[0037] In some embodiments, independently encoding the first image stream feature corresponding to the MJPEG image stream feature set to obtain a first encoded image stream feature includes: determining the first image stream feature corresponding to the MJPEG image stream feature set as the target image stream feature; performing feature extraction optimization on the target image stream feature to obtain an optimized image stream feature; performing feature compression on the optimized image stream feature to obtain a compressed image stream feature; and performing parameter quantization on the compressed feature parameter set corresponding to the compressed image stream feature to obtain the first encoded image stream feature.

[0038] It should be noted that the present invention further elaborates in detail the process of independently encoding the first image stream feature in the MJPEG image stream feature set to ensure that the first image stream feature can be efficiently encoded and key information is retained. During the independent encoding process, first, the first image stream feature is determined as the target image stream feature, and then a series of operations such as feature extraction optimization, feature compression, and parameter quantization are performed on it. Finally, the first encoded image stream feature is obtained. This process not only considers the integrity of the image features but also reduces the amount of encoded data and improves the encoding efficiency through the optimization and quantization steps.

[0039] Specifically, the target image stream feature refers to the first image stream feature in the MJPEG image stream feature set, which is the starting point of the entire encoding process. Feature extraction optimization is the first step in processing the target image stream feature, aiming to remove redundant information and extract the most valuable parts for image recognition and transmission. Feature compression further compresses the optimized image stream feature to reduce the storage space and transmission bandwidth requirements. Parameter quantization processes the compressed feature parameter set by mapping continuous parameter values to a finite number of discrete values to further reduce the data volume. For example, feature extraction optimization can be achieved through a convolutional neural network (CNN) in deep learning. By inputting the target image stream feature map and performing operations such as convolutional layers and activation functions, key features are extracted. Feature compression can be achieved through methods such as principal component analysis (PCA) to map high-dimensional features to a low-dimensional space. Parameter quantization can be achieved by calculating the dynamic range value and scaling factor of the feature parameters and mapping the parameter values to a preset discrete range. These steps together ensure the efficient processing of the first image stream feature during the encoding process.

[0040] Preferably, during the feature extraction optimization process, an intermediate fully-connected layer can be added before the fully-connected layer of the target image stream features. The role of the intermediate fully-connected layer is to further reduce the dimension of the features while retaining key information. For example, assuming the dimension of the original input features is 1024, after being processed by the intermediate fully-connected layer, the dimension of the intermediate output features can be reduced to 256, while the dimension of the final output features remains unchanged. This dimension reduction not only reduces the computational amount but also improves the efficiency of the model. During the parameter quantization process, the dynamic range value between the maximum and minimum values in the compressed feature parameter set can be calculated, and then the scaling factor and offset can be determined according to this dynamic range value. Specifically, the scaling factor is used to map the feature parameter values to a preset discrete range, and the offset is used to adjust the mapped parameter values to ensure they are within an appropriate range. Through these optimization steps, the encoding efficiency and quality of the first image stream features can be further improved.

[0041] In some embodiments, the optimizing the feature extraction of the target image stream features to obtain optimized image stream features includes: adding an intermediate fully-connected layer that meets preset conditions before the fully-connected layer in the target image stream features to obtain initial image stream features, where the fully-connected layer corresponds to input features and output features, the intermediate fully-connected layer is used to obtain intermediate output features according to the input features, the fully-connected layer is used to obtain updated output features according to the intermediate output features, the dimension of the intermediate output features is smaller than the dimensions of the input features and the output features, and the dimension of the updated output features is equal to the dimension of the output features; replacing the convolutional layer in the initial image stream features with depthwise separable convolution to obtain optimized image stream features.

[0042] By adding an intermediate fully-connected layer before the fully-connected layer of the target image stream features and replacing the convolutional layer in the initial image stream features with depthwise separable convolution, the optimization processing of the target image stream features is realized. This method can not only effectively reduce the dimension of the features and the computational amount but also retain key information and improve the efficiency and quality of feature extraction. Through the optimized image stream features, the subsequent compression and encoding processes can be better supported, thereby realizing efficient data processing and transmission.

[0043] Specifically, the target image stream feature refers to the first image stream feature in the MJPEG image stream feature set, which is the starting point of the entire encoding process. The fully connected layer is a common structure in neural networks, used to map input features to output features. The intermediate fully connected layer is an optimized structure introduced in the present invention. It is located before the original fully connected layer and is used to further process the input features. The input features of the intermediate fully connected layer are the input features of the original fully connected layer, and the dimension of its output features is smaller than that of the input features and the output features of the original fully connected layer. For example, assuming the dimension of the original input features is 1024, the intermediate fully connected layer can reduce the dimension to 512, while the dimension of the final output features remains 256. This dimensionality reduction not only reduces the computational amount but also improves the efficiency of the model. Depthwise separable convolution is an optimized convolution operation that decomposes the standard convolution into two independent operations: depthwise convolution and pointwise convolution. Depthwise convolution performs convolution operations on each channel of the input features separately, while pointwise convolution performs 1×1 convolution on the basis of depthwise convolution to increase the number of channels. Through this decomposition, depthwise separable convolution can significantly reduce the computational amount and the number of parameters while maintaining the effect of the convolution operation.

[0044] Preferably, when constructing the intermediate fully connected layer, its input feature dimension can be set to the input feature dimension of the original fully connected layer, and the output feature dimension can be set to half of the output feature dimension of the original fully connected layer. For example, if the input feature dimension of the original fully connected layer is 1024 and the output feature dimension is 256, the output feature dimension of the intermediate fully connected layer can be set to 512. In the implementation process of depthwise separable convolution, a standard depthwise convolution kernel (such as a 3×3 convolution kernel) can be used to perform depthwise convolution on the input features, and then a 1×1 convolution kernel is used to perform pointwise convolution to restore or adjust the number of channels. For example, for an input feature map, first use a 3×3 depthwise convolution kernel to perform depthwise convolution, and then use a 1×1 convolution kernel to adjust its number of channels from 512 to 256. This optimization method not only reduces the computational amount but also improves the running efficiency of the model. In practical applications, by adjusting the output feature dimension of the intermediate fully connected layer and the parameter settings of depthwise separable convolution, it is possible to adapt to different image features and application scenarios, further optimizing the feature extraction and encoding process.

[0045] In some embodiments, performing parameter quantization on the compressed feature parameter set corresponding to the compressed image stream feature to obtain a first encoded image stream feature includes: determining a dynamic range value as the difference between the compressed feature parameter with the largest corresponding value and the compressed feature parameter with the smallest corresponding value in the compressed feature parameter set corresponding to the compressed image stream feature; determining a scaling factor as the ratio of the dynamic range value to a preset maximum representation value; determining an offset as the ratio of the compressed feature parameter with the smallest corresponding value to the scaling factor; performing a scaling mapping on each compressed feature parameter in the compressed feature parameter set according to the scaling factor and the offset to generate a mapped feature parameter, thereby obtaining a set of mapped feature parameters; and determining the first encoded image stream feature corresponding to the compressed image stream feature according to the set of mapped feature parameters.

[0046] It should be noted that the present invention further elaborates in detail the process of performing parameter quantization on the compressed feature parameter set corresponding to the compressed image stream feature. Through the calculation of the dynamic range value, the scaling factor, and the offset, continuous feature parameter values are mapped to finite discrete values, thereby achieving effective compression and quantization of the feature parameters. This method can not only reduce the amount of data after encoding but also ensure that the feature parameters can be accurately restored during the decoding process, thus guaranteeing the image quality.

[0047] Specifically, the compressed image stream feature refers to the image stream feature after feature extraction optimization and compression processing. The compressed feature parameter set is a parameter set related to the compressed image stream feature, and these parameters describe the key information of the image features. The dynamic range value refers to the difference between the maximum value and the minimum value in the compressed feature parameter set, which reflects the range of the feature parameters. The scaling factor is the ratio of the dynamic range value to a preset maximum representation value, and is used to map the feature parameter values to a preset discrete range. The offset is the ratio of the compressed feature parameter with the smallest corresponding value to the scaling factor, and is used to adjust the mapped parameter values to ensure that they are within a suitable range. The set of mapped feature parameters is the parameter set after scaling mapping, and these parameters will be used to generate the final encoded image stream feature. Through the calculation and mapping of these parameters, effective quantization of the image stream feature can be achieved, reducing the amount of data while retaining the key information.

[0048] Preferably, during the parameter quantization process, the following steps can be performed. First, calculate the dynamic range value between the maximum and minimum values in the compressed feature parameter set. For example, assuming the maximum value in the feature parameter set is 100 and the minimum value is 10, the dynamic range value is 90. Then, calculate the scaling factor based on the dynamic range value and the preset maximum representation value. If the preset maximum representation value is 255 (e.g., in 8-bit quantization), the scaling factor is 90 / 255. Next, calculate the offset, which is the ratio of the minimum feature parameter value to the scaling factor, e.g., the offset is 10 / (90 / 255). Finally, use the scaling factor and offset to perform a scaling mapping on each compressed feature parameter to generate the mapped feature parameter set. For example, for a feature parameter value of 50, its mapped value is (50 - 10) / (90 / 255). Through this quantization method, continuous feature parameter values can be mapped to finite discrete values, thus achieving effective data compression and transmission. This method not only reduces the amount of encoded data but also ensures that the feature parameters can be accurately restored during the decoding process, thereby guaranteeing the image quality.

[0049] In some embodiments, the differential encoding of the current image stream features to obtain the initial encoded image stream features includes: determining the difference feature map between the current image stream features and the corresponding previous image stream features; determining the feature parameter difference value between the current image stream features and the corresponding previous image stream features as the difference feature parameter value to obtain the difference feature parameter value set; and determining the difference feature map and the difference feature parameter value set as the initial encoded image stream features corresponding to the current image stream features.

[0050] Differential encoding is an efficient encoding technique. By calculating the difference between the current image stream features and the previous image stream features, only the difference part is encoded, thereby reducing the amount of encoded data. This method is particularly suitable for image streams with little change between consecutive frames, and can effectively reduce the encoding complexity and transmission bandwidth requirements. In the present invention, the initial encoded image stream features generated by differential encoding include the difference feature map and the difference feature parameter value set. These two parts jointly describe the changes between the current frame and the previous frame, providing an optimized basis for subsequent video encoding and transmission.

[0051] Specifically, the current image stream feature refers to any image stream feature in the MJPEG image stream feature set except the first one. The previous image stream feature refers to the previous frame of the image stream feature of the current image stream feature in the time series. The difference feature map is obtained by comparing the image stream feature maps of the current image stream feature and the previous one, which reflects the visual content difference between two frames of images. The difference feature parameter value set is obtained by calculating the difference of the feature parameters between the current image stream feature and the previous one, and these differences describe the changes between two frames of images at the parameter level. For example, if a certain parameter value of the current image stream feature is 50 and the corresponding parameter value of the previous image stream feature is 30, the difference feature parameter value is 20. In this way, differential coding can accurately capture the changes between two frames, so that only these changed parts are encoded in the subsequent coding process, greatly reducing the amount of encoded data.

[0052] Preferably, when implementing differential coding, the following steps can be adopted to further optimize the coding process. First, compare the current image stream feature and the previous one pixel by pixel or region by region to generate a difference feature map. For example, a difference feature map can be generated by calculating the brightness difference, color difference, etc. of the corresponding pixel points of two frames of images. Then, compare the feature parameters of the two frames of images item by item to calculate the difference feature parameter value set. For example, if the feature parameters include brightness, contrast, and color distribution, etc., calculate the differences of these parameters respectively. Then, perform further optimization processing on the difference feature map and the difference feature parameter value set. For example, through threshold judgment, only the difference parts exceeding a certain threshold are retained for encoding to further reduce the data volume. Finally, combine the optimized difference feature map and the difference feature parameter value set into the initial encoded image stream feature corresponding to the current image stream feature. Through these refined steps, differential coding can be more efficiently implemented, improving the coding efficiency and transmission performance.

[0053] The above embodiments of the present invention have the following beneficial effects: The present invention can significantly improve the transmission efficiency of the cloud real machine screen image. By capturing the image in real time through the MJPEG protocol and extracting the MUPEG image stream feature set, combined with feature compression and differential coding technologies, the data transmission volume can be reduced. When the capacity of the encoded feature set is small, embedded transmission is adopted, and when it is large, separated transmission is adopted. This intelligent transmission strategy can adapt to different network conditions and optimize the bandwidth utilization rate. Through feature extraction optimization and parameter quantization processing, the consumption of computing resources can be reduced while ensuring the image quality, and the coding efficiency can be improved.

[0054] Replacing with depthwise separable convolutions and optimizing the intermediate fully connected layers can simplify the neural network structure and reduce the computational complexity of feature extraction. The use of dynamic range values and scaling factors can improve the accuracy of parameter quantization, while the generation of differential feature maps can further reduce redundant data. These technical means working together can achieve efficient and low-latency transmission of the graphical user interface on mobile devices, providing a smooth experience for remote operation and real-time preview.

[0055] As Figure 2 shown, a graphical interface preview device based on the MJPEG protocol in some embodiments, the device includes: a first video encoding unit 201, configured to capture the graphical interface of the mobile phone screen in the cloud real machine in real time based on the MJPEG protocol, obtain the screen graphical interface stream and determine the corresponding MJPEG image stream feature set, wherein the number of image stream features in the MJPEG image stream feature set is the same as the number of frames of the screen graphical interface stream, and the image stream features include image stream feature maps and corresponding feature parameter sets; a compression encoding unit 202, configured to perform compression encoding on each image stream feature in the MJPEG image stream feature set to obtain an encoded image stream feature set.

[0056] A second video encoding unit 203, configured to perform video encoding on the screen graphical interface stream according to the encoded image stream feature set to obtain a video bitstream; an adding unit 204, configured to add the encoded image stream feature set to the video bitstream in response to the capacity of the encoded image stream feature set being less than or equal to a preset threshold to obtain an embedded image stream bitstream; an embedded transmission unit 205, configured to perform embedded transmission on the embedded image stream bitstream, wherein the embedded image stream transmission identifier in the embedded image stream bitstream is 1; a separate transmission unit 206, configured to perform separate transmission on the video bitstream and the encoded image stream feature set in response to the capacity of the encoded image stream feature set being greater than the preset threshold, wherein the embedded image stream transmission identifier in the video bitstream is 0.

[0057] It can be understood that the various modules described in the graphical interface preview device based on the MJPEG protocol correspond to the respective steps in the graphical interface preview method based on the MJPEG protocol described in the reference Figure 1 description. Thus, the operations, features, and beneficial effects described above for the graphical interface preview method based on the MJPEG protocol also apply to the graphical interface preview device based on the MJPEG protocol and the modules included therein, and will not be elaborated here.

[0058] In some embodiments, the compression encoding unit includes: an independent encoding subunit, configured to independently encode the image stream feature corresponding to the first one in the MJPEG image stream feature set to obtain a first encoded image stream feature; a differential encoding subunit, configured to perform differential encoding on each image stream feature in the MJPEG image stream feature set except the first one to obtain an initial encoded image stream feature set; and a feature identifier setting subunit, configured to set a corresponding updated image stream identifier.

[0059] It should be noted that the present invention proposes a graphical interface preview device based on the MJPEG protocol. This device realizes the efficient preview of the mobile phone screen graphical interface in the cloud real machine through the collaborative work of multiple functional units. The core of this device lies in modules such as the first video encoding unit, compression encoding unit, second video encoding unit, addition unit, embedded transmission unit, and separate transmission unit, which perform operations such as real-time capture, compression encoding, video encoding, transmission identifier addition, and transmission control on the screen graphical interface stream. This modular design makes the entire preview process more efficient and flexible, and can be adjusted and optimized according to different application scenarios and requirements.

[0060] Specifically, the first video encoding unit is responsible for real-time capturing the mobile phone screen graphical interface in the cloud real machine based on the MJPEG protocol and determining the corresponding MJPEG image stream feature set. The compression encoding unit then performs compression encoding on each image stream feature in the MJPEG image stream feature set to generate an encoded image stream feature set. The second video encoding unit performs video encoding on the screen graphical interface stream according to the encoded image stream feature set to obtain a video bitstream. The addition unit adds the encoded image stream feature set to the video bitstream to form an embedded image stream bitstream when the capacity of the encoded image stream feature set is less than or equal to a preset threshold. The embedded transmission unit is responsible for transmitting the embedded image stream bitstream, and the separate transmission unit transmits the video bitstream and the encoded image stream feature set separately when the capacity of the encoded image stream feature set is greater than the preset threshold. These units work together to ensure the efficient processing and transmission of the image stream. For example, the preset threshold can be set according to network bandwidth and image quality requirements. When the capacity of the encoded image stream feature set is less than or equal to this threshold, using embedded transmission can reduce transmission latency; while when the capacity is greater than the threshold, separate transmission can avoid data loss and transmission errors.

[0061] Preferably, when implementing the device, the functions and operations of each unit can be further refined. For example, the compression encoding unit may include an independent encoding subunit, a differential encoding subunit, and a feature identification setting subunit. The independent encoding subunit is responsible for independently encoding the first image stream feature in the MJPEG image stream feature set to generate a first encoded image stream feature. The differential encoding subunit then performs differential encoding on subsequent image stream features to generate an initial encoded image stream feature set. The feature identification setting subunit is used to set an updated image stream identifier to mark whether each frame of the image stream feature has been updated. In practical applications, the functions of these units can be implemented by software or hardware. For example, a dedicated image processing chip can be used to implement the functions of the video encoding and compression encoding units to improve processing speed and efficiency. At the same time, the parameter settings of each unit can be adjusted according to actual needs, such as adjusting preset thresholds, parameters of encoding algorithms, etc., to achieve the best preview effect and transmission performance.

[0062] Reference is made below to Figure 3 , which shows a schematic structural diagram of an electronic device 300 suitable for use in implementing some embodiments of the present invention. The electronic device in some embodiments of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The terminal device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0063] As Figure 3 shown, the electronic device 300 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0064] Typically, the following devices can be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. The communication device 309 can allow the electronic device 300 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 3 the electronic device 300 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices can be implemented or had. Figure 3 Each block shown in can represent one device or, as needed, multiple devices.

[0065] Further, the storage medium of the embodiments of the present application stores program instructions capable of implementing all the above methods. Among them, the program instructions can be stored in the above storage medium in the form of a software product, including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. And the foregoing storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, or terminal devices such as computers, servers, mobile phones, and tablets.

[0066] The above description is only some preferred embodiments of the present invention and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features having similar functions disclosed in the embodiments of the present invention.

Claims

1. A graphical interface preview method based on MJPEG protocol, characterized in that: include: Based on the MJPEG protocol, the screen graphic interface of the mobile phone in the cloud real machine is captured in real time to obtain the screen graphic interface stream and determine the corresponding MJPEG image stream feature set, wherein the number of image stream features in the MJPEG image stream feature set is the same as the number of frames of the screen graphic interface stream, and the image stream feature includes an image stream feature map and a corresponding feature parameter set; each image stream feature in the MJPEG image stream feature set is compressed and encoded to obtain an encoded image stream feature set; according to the encoded image stream feature set, the screen graphic interface stream is video encoded to obtain a video bit stream; in response to the capacity of the encoded image stream feature set being less than or equal to a preset threshold, the encoded image stream feature set is added to the video bit stream to obtain an embedded image stream bit stream; the embedded image stream bit stream is embeddedly transmitted, wherein the embedded image stream transmission identifier in the embedded image stream bit stream is 1; in response to the capacity of the encoded image stream feature set being greater than a preset threshold, the video bit stream and the encoded image stream feature set are transmitted respectively, wherein the embedded image stream transmission identifier in the video bit stream is 0.

2. The method according to claim 1, characterized in that The method of compressing and encoding each image stream feature in the MJPEG image stream feature set to obtain the encoded image stream feature set includes: obtaining an updated image stream identifier list, wherein the number of updated image stream identifiers in the updated image stream identifier list is the same as the number of frames of the screen graphic interface stream; independently encoding the image stream feature corresponding to the first one in the MJPEG image stream feature set to obtain a first encoded image stream feature, and setting the corresponding updated image stream identifier to 1; performing the following encoding steps in sequence on each image stream feature except the first one in the MJPEG image stream feature set to obtain an initial encoded image stream feature set: in response to the current image stream feature being different from the previous image stream feature, differentially encoding the current image stream feature to obtain an initial encoded image stream feature, and setting the corresponding updated image stream identifier to 1; in response to the current image stream feature being the same as the previous image stream feature, setting the corresponding updated image stream identifier to 0; and determining the first encoded image stream feature and the initial encoded image stream feature set as the encoded image stream feature set.

3. The method according to claim 2, characterized in that The method of independently encoding the first image stream feature corresponding to the MJPEG image stream feature set to obtain the first encoded image stream feature includes: determining the first image stream feature corresponding to the MJPEG image stream feature set as the target image stream feature; performing feature extraction optimization on the target image stream feature to obtain an optimized image stream feature; performing feature compression on the optimized image stream feature to obtain a compressed image stream feature; and performing parameter quantization on a compressed feature parameter set corresponding to the compressed image stream feature to obtain the first encoded image stream feature.

4. The method according to claim 3, characterized in that The feature extraction optimization of the target image flow feature to obtain the optimized image flow feature includes: adding an intermediate fully connected layer that meets preset conditions before the fully connected layer in the target image flow feature to obtain the initial image flow feature, wherein the fully connected layer corresponds to an input feature and an output feature, the intermediate fully connected layer is used to obtain an intermediate output feature according to the input feature, the fully connected layer is used to obtain an updated output feature according to the intermediate output feature, the dimension of the intermediate output feature is smaller than the dimension of the input feature and the output feature, and the dimension of the updated output feature is equal to the dimension of the output feature; the convolution layer in the initial image flow feature is replaced by a depth-separable convolution to obtain the optimized image flow feature.

5. The method according to claim 3, characterized in that: The method of performing parameter quantization on a compressed feature parameter set corresponding to the compressed image stream feature to obtain a first coded image stream feature comprises: determining the difference between a compressed feature parameter with a maximum corresponding value and a compressed feature parameter with a minimum corresponding value in the compressed feature parameter set corresponding to the compressed image stream feature as a dynamic range value; determining a ratio of the dynamic range value to a preset maximum representation value as a scaling factor; determining a ratio of the compressed feature parameter with a minimum corresponding value to the scaling factor as an offset; performing scaling mapping on each compressed feature parameter in the compressed feature parameter set according to the scaling factor and the offset to generate a mapped feature parameter to obtain a mapped feature parameter set; and determining, according to the mapped feature parameter set, a first coded image stream feature corresponding to the compressed image stream feature.

6. The method according to claim 2, characterized in that The differential encoding of the current image stream feature to obtain the initial encoded image stream feature includes: determining a difference feature map between the current image stream feature and the corresponding previous image stream feature; determining the feature parameter difference between the current image stream feature and the corresponding previous image stream feature as a difference feature parameter value to obtain a difference feature parameter value set; and determining the difference feature map and the difference feature parameter value set as the initial encoded image stream feature corresponding to the current image stream feature.

7. A graphical interface preview device based on the MJPEG protocol, characterized in that: include: The first video encoding unit is configured to capture the screen graphic interface of the mobile phone in the cloud real machine in real time based on the MJPEG protocol, obtain the screen graphic interface stream and determine the corresponding MJPEG image stream feature set, wherein the number of image stream features in the MJPEG image stream feature set is the same as the number of frames of the screen graphic interface stream, and the image stream feature includes an image stream feature map and a corresponding feature parameter set; the compression encoding unit is configured to compress and encode each image stream feature in the MJPEG image stream feature set to obtain an encoded image stream feature set; the second video encoding unit is configured to visually encode the screen graphic interface stream according to the encoded image stream feature set. The invention relates to a method for transmitting the encoded image stream feature set to the video bitstream by frequency encoding to obtain a video bitstream; an adding unit is configured to, in response to the capacity of the encoded image stream feature set being less than or equal to a preset threshold, add the encoded image stream feature set to the video bitstream to obtain an embedded image stream bitstream; an embedded transmission unit is configured to perform embedded transmission on the embedded image stream bitstream, wherein the embedded image stream transmission identifier in the embedded image stream bitstream is 1; and a separate transmission unit is configured to, in response to the capacity of the encoded image stream feature set being greater than a preset threshold, transmit the video bitstream and the encoded image stream feature set separately, wherein the embedded image stream transmission identifier in the video bitstream is 0.

8. The device according to claim 7, characterized in that The compression coding unit includes: an independent coding subunit, configured to independently encode the image stream feature corresponding to the first one in the MJPEG image stream feature set to obtain a first encoded image stream feature; a differential coding subunit, configured to differentially encode each image stream feature except the first one in the MJPEG image stream feature set to obtain an initial encoded image stream feature set; and a feature identifier setting subunit, configured to set a corresponding updated image stream identifier.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images