A desktop sharing and mapping method for an Android system

Abstract

The application provides a desktop sharing and mapping method for an Android system, which is used for solving the ghosting problem of a remote conference system when mapping under the Android system, and comprises the following steps: S101, a monitoring service is created, and the monitoring service is configured to monitor mapping data of a DIV layer; S102, when the monitoring service monitors the mapping data, a screen capture module sets coordinate values of a region corresponding to the mapping data in the screen data collected by the screen capture module to 0, and fills the region with the coordinate values set to 0 by using a preset filling algorithm; and S103, the screen capture module encodes the screen data processed in the step S102, and forwards the screen data to the outside after encapsulation by using a preset protocol. The application can improve the readability of data in the desktop sharing and mapping process, and further improve the efficiency of the remote conference.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of online mapping technology of video conference, and particularly relates to a desktop sharing and mapping method for an Android system. BACKGROUND

[0002] Remote collaborative office work solves the problem of inconvenient communication of off-site office work by remote conference application. When off-site multiple people consult or hold a meeting, they usually communicate their opinions by marking and sharing. A commonly used method is to use a conference collaboration whiteboard to complete the mapping function. In order to facilitate users to hold a remote conference, remote conference systems with mapping functions have been developed on Windows, iOS, Android and other platforms. Among them, the Android system is a typical multi-user single-application system (only one application can be operated at the same time), and the remote conference system on the platform will have a ghosting problem when real-time collaboration and mapping are needed. For example, in a remote conference scenario, A is the host, and B, C and D are participants. When A, B, C and D four users simultaneously perform real-time mapping on the remote conference system running on the Android system, if B performs a mapping operation, the mapping data of B will be forwarded to the screen display of A locally. The remote conference system will collect the screen display content of A locally and share it with the participants. As a participant, B will see his own mapping strokes and A's screen content with his own mapping strokes on the local screen at the same time, which will cause ghosting and affect the participants' view of the screen sharing content. Due to the underlying design of the Android system, the screen content and mapping data cannot be filtered to solve the ghosting problem. SUMMARY

[0003] In view of this, the present application aims to provide a desktop sharing and mapping method for an Android system, which can solve the ghosting problem when desktop sharing and mapping are performed on the Android system, improve data readability, and ensure the viewing experience of users.

[0004] To achieve the above-mentioned application purpose, the present application provides a desktop sharing and mapping method for an Android system, which is applied to a desktop sharing and mapping application running on an Android system, and comprises the following steps:

[0005] S101, creating a monitoring service, and configuring the monitoring service to monitor the mapping data of the DIV layer;

[0006] S102, when the monitoring service monitors the mapping data, a screen capture module sets the coordinate values of the area corresponding to the mapping data in the screen data collected by the screen capture module to 0, and fills the area with the coordinate values set to 0 using a preset filling algorithm;

[0007] S103, the screen capture module encodes the screen data processed in step S102, encapsulates through a preset protocol, and forwards to the outside.

[0008] Further, the desktop sharing mapping application is configured to perform the following steps:

[0009] S201, determining whether the current user of the desktop sharing mapping application is a first type of user or a second type of user.

[0010] S202, when the current user is a first type of user, the screen capture module collects local screen data, encodes the screen data, encapsulates through a preset protocol, and sends to a streaming media server, and a DIV layer pushes mapping data of the layer to a mapping server in JSON form.

[0011] S203, when the current user is a second type of user, the DIV layer pushes the mapping data of the layer to the mapping server in JSON form.

[0012] Further, after receiving the encapsulated screen data, the streaming media server transmits the screen data to the desktop sharing mapping application corresponding to each second type of user, the mapping server transmits the mapping data to the desktop sharing mapping application corresponding to other first or second type of users who are not the mapping data uploaders, and the desktop sharing mapping application renders according to the screen data and the mapping data after receiving the screen data and the mapping data, and generates screen display content.

[0013] Further, a preset filling algorithm is used to fill the area where the coordinate value is set to 0, which specifically includes the following steps:

[0014] S301, positioning the to-be-filled area and identifying the boundary of the to-be-filled area;

[0015] S302, analyzing the surrounding area of the to-be-filled area to find the optimal matching pixel or pixel block;

[0016] S303, based on the optimal matching pixel or pixel block, layer-by-layer filling and smooth transition are performed on the to-be-filled area.

[0017] Further, step S302 specifically includes the following steps:

[0018] S401, finding the optimal matching pixel of the surrounding area of the to-be-filled area from multiple dimensions, the dimensions including color, structure, texture, brightness, and contrast;

[0019] S402, selecting a pixel block most similar to the internal boundary from the directly adjacent area of the to-be-filled area for copying.

[0020] Further, the step S402 of searching the optimal matching pixel block specifically comprises: calculating the distance between the boundary of the to-be-filled region and the pixel block of the surrounding environment by a distance measurement algorithm, and searching the pixel block directly adjacent to the boundary of the to-be-filled region.

[0021] Further, the step S402 specifically comprises the following steps:

[0022] S501, selecting an image region from the periphery of the to-be-filled region as a template;

[0023] S502, sliding the template in a larger image region, and calculating the similarity between the template and the pixel block at each position covered by the template in the sliding process;

[0024] S503, outputting the pixel block at the position with the highest similarity as the result.

[0025] Further, in the step S303, the to-be-filled region is progressively filled based on the optimal matching pixel or pixel block, and after each filling, the filled region is locally smoothed.

[0026] Further, the region with the coordinate value set to 0 is filled by using a preset filling algorithm, and the preset filling algorithm specifically comprises the following steps:

[0027] S601, training a deep learning model;

[0028] S602, extracting key features of a complete region around the to-be-filled region, and taking the key features as the input of the deep learning model;

[0029] S603, the deep learning model generates new pixels or pixel blocks based on the input key features;

[0030] S604, post-processing the pixels or pixel blocks generated in the previous step, and filling the pixels or pixel blocks after the post-processing into the to-be-filled region.

[0031] Further, the post-processing comprises style fusion processing and color adjustment processing.

[0032] Compared with the prior art, the present application has the following advantages:

[0033] The present application aims at the ghosting problem in the plotting of a remote conference system under an Android system, and provides a desktop sharing and plotting method for an Android system, which creates a monitoring service to monitor the plotting data of a DIV layer, and when the monitoring service monitors the plotting data, a screen capture module sets the coordinate values of the area corresponding to the plotting data in the screen data collected by the screen capture module to 0, and then fills the area with the coordinate values set to 0 using a preset filling algorithm, so that the screen content captured by the screen capture module removes the plotting content, thereby solving the ghosting problem and improving the readability of data and the efficiency of remote conference. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only preferred embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0035] Figure 1 is a schematic diagram of the overall process of a desktop sharing and plotting method for an Android system provided by an embodiment of the present application.

[0036] Figure 2 is a schematic diagram of the data transmission of a desktop sharing and plotting application provided by an embodiment of the present application.

[0037] Figure 3 is a schematic diagram of the work flow of a desktop sharing and plotting application provided by an embodiment of the present application.

[0038] Figure 4 is a schematic diagram of the area filling process of a preset filling algorithm provided by an embodiment of the present application.

[0039] Figure 5 is a schematic diagram of the area filling process of a preset filling algorithm provided by another embodiment of the present application. DETAILED DESCRIPTION

[0040] The principles and characteristics of the present application are described below in combination with the drawings, and the listed embodiments are only used to explain the present application, and are not used to limit the scope of the present application.

[0041] REFERENCE Figure 1 The present embodiment provides a desktop sharing and plotting method for an Android system, which is applied to a desktop sharing and plotting application, runs on an Android system, and includes the following steps:

[0042] S101, a monitoring service is created, and the monitoring service is configured to monitor the plotting data of a DIV layer.

[0043] S102, when the monitoring service detects the mapping data, the screen capture module sets the coordinate values of the region corresponding to the mapping data in the screen data collected by the screen capture module to 0, and fills the region whose coordinate values are set to 0 using a preset filling algorithm.

[0044] S103, the screen capture module encodes the screen data processed in step S102, and forwards the screen data after being encapsulated by a preset protocol.

[0045] With reference to Figure 2 and Figure 3 In the embodiment, the desktop sharing and mapping application respectively communicates with a streaming media server and a mapping server, wherein the streaming media server is responsible for processing and forwarding screen data, i.e. desktop content to be shared; and the mapping server is responsible for processing and forwarding mapping data, i.e. graphics, text and other content drawn by participants. The desktop sharing and mapping application is configured to perform the following steps:

[0046] S201, determine whether the current user of the desktop sharing and mapping application is a first type of user or a second type of user. In the embodiment, the first type of user is a user initiating remote desktop sharing, i.e. a conference host; and the second type of user is a user participating in remote desktop sharing, i.e. a participant.

[0047] S202, when the current user is the first type of user, the screen capture module collects local screen data, encodes the screen data, and sends the screen data to the streaming media server after being encapsulated by a preset protocol; and the DIV layer pushes the mapping data of the layer to the mapping server in JSON format.

[0048] In the embodiment, the desktop display content of the first type of user needs to be shared to all participants, and therefore when the current user of the desktop sharing and mapping application is the first type of user, the screen capture module collects local screen data, and sends the screen data to the streaming media server after being encoded and encapsulated to forward the screen data to the desktop sharing and mapping applications corresponding to other participants.

[0049] S203, when the current user is the second type of user, the DIV layer pushes the mapping data of the layer to the mapping server in JSON format. The second type of user can perform mapping operations in the DIV layer during remote desktop sharing, and the DIV layer sends the recorded mapping data to the mapping server, so that the mapping server can synchronize the mapping data to the desktop sharing and mapping applications corresponding to other participants to realize shared display of the mapping data.

[0050] In this embodiment, the desktop sharing mapping application shows information to the user and responds to user operations through the front-end presentation layer. The layer used to show screen data in the front-end presentation layer is called the screen content layer, and the DIV layer is a transparent layer placed above the screen content layer, which is a key part of the front-end presentation layer and plays an important role in showing and operating mapping content in the user interface. Through close cooperation with the application logic layer (a layer used to handle the core business logic of the application program, such as handling user mapping operations, executing commands, managing local state, etc.), it can respond to user operations, show real-time updated content, and share these contents with other users through the data synchronization layer (a layer used to synchronize the operation and data state between different users, ensuring that all users can see the latest shared content). Such a layered architecture not only facilitates the coordinated work of various parts of the system, but also improves the maintainability and scalability of the entire system.

[0051] The streaming server transmits the screen data to the desktop sharing mapping applications corresponding to the second type of users after receiving the encapsulated screen data. The mapping server transmits the mapping data to the desktop sharing mapping applications corresponding to the first type of users or the second type of users other than the mapping data uploading party after receiving the mapping data. The desktop sharing mapping application renders the screen data and the mapping data after receiving the screen data and the mapping data, and generates screen display content.

[0052] As an optional implementation, referring to Figure 4 , a preset filling algorithm is used to fill the area where the coordinate value is set to 0, which specifically includes the following steps:

[0053] S301, positioning the to-be-filled area and identifying the boundary of the to-be-filled area. The boundary of the to-be-filled area can be derived from the coordinate value of the mapping data.

[0054] S302, analyzing the surrounding area of the to-be-filled area to find the optimal matching pixel or pixel block.

[0055] S303, based on the optimal matching pixel or pixel block, the to-be-filled area is filled layer by layer and smoothly transitioned. By gradually filling from the edge to the center of the to-be-filled area, it is ensured that each step of filling the pixel can be naturally integrated with the surrounding pixels, avoiding obvious visual discontinuity.

[0056] As a further optional implementation, step S302 specifically includes the following steps:

[0057] S401, find the optimal matching pixel of the surrounding area of the to-be-filled area from multiple dimensions, including but not limited to color, structure, texture, brightness, and contrast.

[0058] Among them, color is the primary factor for judging the matching degree of pixels, because the consistency of color will directly affect the visual harmony of the image. For example, the similarity between the candidate pixel and the surrounding pixel of the to-be-filled region can be judged by calculating the color distance between the candidate pixel and the surrounding pixel of the to-be-filled region, as one of the evaluation indexes of the optimal matching pixel. The calculation of color distance can be realized by Euclidean distance or other color similarity indexes (such as hue, saturation, brightness difference) in color space such as RGB, HSV, etc.

[0059] Texture is defined by the pattern or arrangement of pixels that repeatedly appear in the image, and the continuity of texture is very important for the overall naturalness of the completed image. For example, the texture features can be analyzed and matched by using gray level co-occurrence matrix (GLCM), local binary pattern (LBP), etc. The primary principle of matching is to find the pixel or pixel block that is closest to the texture features of the surrounding region.

[0060] Structural information reflects the geometric and spatial relationship of the image, including edge direction, shape, etc. For example, the structural similarity between the surrounding region of the to-be-filled region and the candidate pixel block can be evaluated and compared by using structural similarity index (SSIM) or edge detection algorithm.

[0061] The consistency of brightness and contrast helps to ensure the consistency of the filled region with the original image, so as to avoid obvious visual discontinuity. For example, by measuring and comparing the brightness level and contrast of the surrounding region of the to-be-filled region and the candidate pixel, the brightness and contrast of the candidate pixel can be adjusted to better match.

[0062] S402, selecting the pixel block closest to the internal boundary from the directly adjacent region of the to-be-filled region for copying.

[0063] As a further optional implementation, the step S402 of finding the optimal matching pixel block specifically calculates the distance between the boundary of the to-be-filled region and the pixel block of the surrounding environment by using distance measurement algorithm, and finds the pixel block directly adjacent to the boundary of the to-be-filled region. For example, the distance measurement algorithm can use Euclidean distance, Manhattan distance, Mahalanobis distance or other distance measurement algorithms. When the to-be-filled region needs to be filled, the filling is performed by searching for other pixels closest to the surrounding pixels of the to-be-filled region in the data structure. In a multi-dimensional space (such as RGB color space), each pixel can be regarded as a point. Data structures such as k-d tree, ball tree or LSH (locality-sensitive hashing) can be used to effectively organize and search these points.

[0064] In another optional implementation, the step S402 specifically includes the following steps:

[0065] S501. Select a template from the periphery of the region to be filled. The template represents a small region of the target image part that we want to find, which is usually from the image itself that needs to be copied or find similar parts.

[0066] S502. Slide the template in the larger image region, and for each position covered by the template during the sliding, calculate the similarity between the template and the pixel block at that position.

[0067] S503. Output the pixel block at the position with the highest similarity as the result, indicating that the most similar region to the template has been found in the large image. The similarity calculation in this step can be measured by cross-correlation, mean squared error, or normalized cross-correlation, etc.

[0068] In step S303, the region to be filled is progressively filled based on the optimally matched pixels or pixel blocks, and after each filling, the filled region is locally smoothed.

[0069] For example, the local smoothing process can be a blur process or a resampling process. The purpose of the blur process is to replace each pixel in the image with some kind of average of its surrounding pixels, thereby reducing the high-frequency components of the image. The blur process can use Gaussian blur, mean blur, or other blur processing methods. Resampling involves changing the size of the image, which requires estimating the color value of the new pixel position by interpolation method, and using the nearest neighbor interpolation method to resample.

[0070] As another optional implementation, with reference to Figure 5 , the region with the coordinate value set to 0 is filled using a preset filling algorithm, which includes the following steps:

[0071] S601. Train the deep learning model. In this embodiment, the deep learning model selected is ResNet, and it can be understood that those skilled in the art can also select other deep learning models for training according to actual needs.

[0072] S602. Extract the key features of the complete region around the region to be filled, and use the key features as the input of the deep learning model. For example, the key features can be texture, color, structural shape, etc.

[0073] S603. The deep learning model generates new pixels or pixel blocks based on the input key features.

[0074] S604, post-processing the pixels or pixel blocks generated in the previous step, and filling the post-processed pixels or pixel blocks into the area to be filled.

[0075] Exemplarily, the training of the deep learning model includes the following steps:

[0076] First step, data preparation:

[0077] S701, data collection: according to the purpose of the model, a large number of related image data are collected.

[0078] S702, data cleaning: delete damaged, unclear or irrelevant images, ensure data quality.

[0079] S703, data augmentation: increase the diversity of data by rotating, scaling, cropping, color adjustment, etc., and improve the generalization ability of the model.

[0080] S704, label data: for supervised learning, the data need to be labeled, that is, to provide the correct output for the image (such as marking the missing area in the image).

[0081] S705, split data: usually divide the data into training set, validation set and test set, respectively for training model, adjusting parameters and evaluating model.

[0082] Second step, configure training parameters, the purpose is to set the hyperparameters in the training process, including:

[0083] Learning rate: determines the speed of updating model parameters, too high may lead to unstable learning, too low may lead to slow training speed.

[0084] Batch size: affects the stability of training and memory usage, needs to be adjusted according to available computing resources.

[0085] Loss function: selected according to the specific type of problem, for example, mean square error may be used for regression task, cross entropy for classification task.

[0086] Optimizer: such as SGD, Adam, etc., determines how to update the network weights.

[0087] Third step, execute training, including the following process:

[0088] Forward propagation: input data through the network, calculate the output.

[0089] Loss calculation: calculate the difference between the output and the real label.

[0090] Back propagation: calculate the gradient according to the loss, and update the weights of the network.

[0091] Iteration Process: Repeat the forward propagation, loss calculation, and backpropagation until the model performance no longer improves or reaches the preset number of iterations.

[0092] Step 4: Evaluation and Optimization, the purpose is to verify the effectiveness and generalization ability of the model, including the following steps:

[0093] Test on validation set: Regularly test the model on the validation set to monitor the performance changes during training.

[0094] Adjust parameters: Adjust training parameters according to validation results, such as adjusting learning rate, changing network structure, etc.

[0095] Final test: Evaluate the model performance on an independent test set to ensure the model has good generalization ability.

[0096] As an optional implementation, the post-processing includes style fusion processing and color adjustment processing.

[0097] Among them, the purpose of style fusion processing is to ensure that different parts of the image look visually coordinated and unified, especially when image synthesis or repair, the newly introduced elements are naturally integrated with the original image background. For example, style fusion processing can be achieved through Gram matrix matching: in the application of deep learning, especially in style transfer, style fusion can be achieved by calculating the Gram matrix of the feature response (through convolutional neural network) of the content image and the style image and minimizing the difference between them; style fusion processing can also be achieved through mixed layers: in image editing software, by using different blending modes (such as soft light, hard light, overlay, etc.), the visual style of one image or layer can be integrated into another, which helps to unify the lighting, shadows and color tones.

[0098] The purpose of color adjustment processing is to unify the color tone, contrast and brightness of the image, so that the entire image looks more natural, especially after introducing new elements in image repair or synthesis. Color adjustment processing includes the following aspects:

[0099] Color balance adjustment: Adjust the color balance in the image, change the color of the shadow, mid-tone and highlight area to match the overall style of the image.

[0100] Curve adjustment: By adjusting the input and output relationship curve of the RGB color channel, the color contrast and brightness level of the image can be finely controlled.

[0101] Tone mapping: In more complex image processing, tone mapping technology can be used to adjust the dynamic range of the image, so that the image obtains the best details and color performance from different brightness areas.

[0102] The above merely describes preferred embodiments of the present application and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for desktop sharing and mapping for Android platform, characterized in that, The method is applied to a desktop sharing mapping application running on an Android platform, and comprises the following steps: S101, a monitoring service is created and configured to monitor mapping data of a DIV layer; S102, when the monitoring service detects the mapping data, a screen capture module sets coordinate values of a region corresponding to the mapping data in the captured screen data to 0, and fills the region with the set coordinate values using a preset filling algorithm; S103, the screen capture module encodes the screen data processed in step S102, and forwards the encoded screen data to an external device after encapsulation according to a preset protocol; The filling of the region with the set coordinate values using the preset filling algorithm specifically comprises the following steps: S301, the region to be filled is located and the boundary of the region to be filled is identified; S302, a surrounding region of the region to be filled is analyzed to find an optimal matching pixel or pixel block; S303, the region to be filled is filled layer by layer based on the optimal matching pixel or pixel block, and a smooth transition is performed. Step S302 specifically comprises the following steps: S401, an optimal matching pixel of the surrounding region of the region to be filled is found from multiple dimensions, including color, structure, texture, brightness and contrast; S402, a pixel block most similar to the internal boundary is selected from a directly adjacent region of the region to be filled for copying; The finding of the optimal matching pixel block in step S402 specifically comprises: the distance between the boundary of the region to be filled and a pixel block of the surrounding environment is calculated by a distance measurement algorithm, and a pixel block directly adjacent to the boundary of the region to be filled is found; or the following steps are performed: S501, an image region is selected as a template from the surrounding of the region to be filled; S502, the template is slid in a larger image region, and the similarity between the template and a pixel block at each position covered by the template in the sliding process is calculated; S503, the pixel block at the position with the highest similarity is output as a result.

2. The desktop sharing and mapping method for Android platform according to claim 1, characterized in that, The desktop sharing mapping application is configured to perform the following steps: S201, it is determined whether a current user of the desktop sharing mapping application is a first type of user or a second type of user; S202, when the current user is the first type of user, a screen capture module captures local screen data, encodes the screen data, and sends the encoded screen data to a streaming media server after encapsulation according to a preset protocol; and the DIV layer pushes mapping data of the layer to a mapping server in JSON format; S203, when the current user is the second type of user, the DIV layer pushes the mapping data of the layer to the mapping server in JSON format.

3. The desktop sharing and mapping method for Android platform according to claim 2, characterized in that, After receiving the encapsulated screen data, the streaming media server transmits the screen data to desktop sharing mapping applications corresponding to each second type of user; after receiving the mapping data, the mapping server transmits the mapping data to desktop sharing mapping applications corresponding to other first type of users or second type of users except the user uploading the mapping data; and after receiving the screen data and the mapping data, the desktop sharing mapping applications render the screen data and the mapping data to generate screen display content.

4. The desktop sharing and mapping method for Android platform according to claim 1, characterized in that, In step S303, the region to be filled is filled gradually based on the optimal matching pixel or pixel block, and local smoothing is performed on the filled region after each filling.

5. The desktop sharing and mapping method for Android platform according to claim 1, characterized in that, The preset filling algorithm is used to fill the region with the coordinate value set to 0, and the specific steps include the following steps: S601, training a deep learning model; S602, extracting key features of a complete region around the to-be-filled region, and taking the key features as inputs of the deep learning model; S603, the deep learning model generates new pixels or pixel blocks based on the input key features; S604, post-processing the pixels or pixel blocks generated in the previous step, and filling the pixels or pixel blocks after the post-processing into the to-be-filled region.

6. The desktop sharing and mapping method for Android platform according to claim 5, characterized in that, The post-processing includes style fusion processing and color adjustment processing.

Images