Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and equipment, and relates to the technical field of data processing, and the method comprises the steps: obtaining region-of-interest data sent by a client; extracting data of a first area of the current frame image according to the attention area data; extracting data of a second area of the current frame image according to the pixel similarity of the current frame image and the adjacent frame image; sending the data of the first area and the data of the second area of the current frame image to the client; the data of the first area and the data of the second area are used for obtaining a complete current frame image in combination with a full frame image periodically sent to the client. According to the embodiment of the invention, the transmission data volume can be reduced, and the watching experience of users is ensured.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, storage medium and electronic device. Background Technology

[0002] Real-Time Cloud Rendering is a technology that shifts graphics rendering computation from terminal devices to cloud servers. It utilizes high-performance computing resources in the cloud (such as GPU clusters) to process complex graphics data in real time, generating dynamic visuals. The rendering results are then transmitted to terminal devices (phones, tablets, PCs, AR / VR headsets, etc.) in a lightweight format (such as frame streams), ultimately achieving a "low-configuration, high-experience" interactive effect on the terminal.

[0003] With the development of real-time cloud rendering services, the data volume of frame image streams is increasing, placing higher demands on network transmission quality. If the network transmission quality cannot meet these requirements, it will lead to a decrease in the playback quality on the client side. Currently, existing technologies use methods such as transmitting the parts of the current frame image that differ from the previous frame image to the client and then stitching the images together on the client side to reduce the amount of data transmitted. However, this method can easily lead to a perceived deterioration in image quality, thus affecting the user's viewing experience. Summary of the Invention

[0004] In view of this, this application provides a data processing method, apparatus, storage medium, and electronic device that can effectively reduce the amount of data transmitted and ensure the user's viewing experience.

[0005] In a first aspect, this application provides a data processing method, including: Obtain the attention area data sent by the client; the attention area data is generated based on the range of the terminal device screen that the human eye is focused on. Based on the region of interest data, extract the data of the first region of the current frame image; Based on the pixel similarity between the current frame image and the adjacent frame images, the data of the second region of the current frame image is extracted; there is a similarity difference between the second region of the current frame image and the second reference region of the adjacent frame images, and the coordinate range of the second region of the current frame image and the second reference region of the adjacent frame images is the same; The system sends data from the first region and the second region of the current frame image to the client. The data from the first region and the second region are combined with the full frame images periodically sent to the client to obtain the complete current frame image.

[0006] Optionally, based on the pixel similarity between the current frame image and adjacent frame images, data of the second region of the current frame image is extracted, including: The non-first region of the current frame image is divided according to preset rules to obtain multiple candidate regions; Pixels within a candidate region are extracted and their similarity to pixels in a reference candidate region of an adjacent frame image is calculated to obtain the first similarity for each candidate region. The reference candidate region corresponds one-to-one with the candidate region and has the same coordinate range. Based on the pixel distance between the reference pixels of the candidate region and the reference pixels of the first region, candidate regions that meet the similarity conditions are selected one by one from small to large until all candidate regions that meet the similarity conditions are selected, or there are candidate regions that do not meet the similarity conditions. Extract candidate regions that meet the similarity criteria, and assign these candidate regions to the second region.

[0007] Optionally, the non-first region of the current frame image is divided according to preset rules to obtain multiple candidate regions, including: Using the origin of the first region as the center, the radius of the candidate region is obtained by using the radius of the first region and the preset step size. The non-first region of the current frame image is divided to obtain multiple non-overlapping candidate regions. The preset step size includes the length of the preset step size and the number of preset step sizes. The radii of adjacent candidate regions differ by the length of the preset step size.

[0008] Optionally, pixel similarity is calculated between pixels within the candidate region and pixels in the reference candidate region of adjacent frame images to obtain the first similarity scores for multiple candidate regions, including: The candidate region is sampled to obtain multiple sub-regions; The pixels of multiple sub-regions included in the candidate region are extracted and their pixel similarity is calculated with the pixels of the reference sub-region of the previous frame image to obtain the second similarity of multiple sub-regions; the reference sub-region corresponds one-to-one with the sub-regions and has the same coordinate range; The first similarity of the candidate region is obtained by averaging the second similarities of multiple sub-regions.

[0009] Optionally, candidate regions that meet the similarity criteria are selected one by one from smallest to largest based on the pixel distance between the candidate regions and the first region, including: The first fusion value is obtained based on the first similarity of the candidate regions and the weight value corresponding to the candidate regions; the weight value corresponding to the candidate regions is determined based on the pixel distance between the candidate regions and the first region. The overall similarity is obtained based on the first fusion value corresponding to the unselected candidate regions; If the overall similarity is less than the preset similarity threshold, then the candidate region that is closest to the pixel distance of the first region is selected from the unselected candidate regions. Return to the step of obtaining the overall similarity based on the first fusion value corresponding to the unselected candidate regions, until there are no unselected candidate regions, or the overall similarity is greater than the preset similarity threshold.

[0010] Optionally, the weight value corresponding to the candidate region is negatively correlated with the pixel distance between the reference pixel of the candidate region and the reference pixel of the first region.

[0011] Optionally, the method also includes: Get the current frame image; If the current frame image is a keyframe image, or if the full frame image has not been sent to the client within the current image update cycle, then all regions of the current frame image will be transmitted to the client.

[0012] Secondly, this application provides a data processing apparatus, comprising: The acquisition module is configured to acquire attention area data sent by the client; the attention area data is generated based on the range of human eye focus on the terminal device screen. The extraction module is configured to extract data of the first region of the current frame image based on the region of interest data; The calculation module is configured to extract data of a second region of the current frame image based on the pixel similarity between the current frame image and the adjacent frame images; the second region of the current frame image has a similarity difference with the second region of the adjacent frame images, and the coordinate range of the second region of the current frame image and the second region of the adjacent frame images is the same; The sending module is configured to send data of the first region and the second region of the current frame image to the client; the data of the first region and the data of the second region are used to combine with the full frame images periodically sent to the client to obtain the complete current frame image.

[0013] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect.

[0014] Fourthly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect.

[0015] Fifthly, this application provides a computer program product having a computer program stored thereon, wherein the computer program product, when executed by a processor, implements the method of the first aspect.

[0016] By employing the above technical solutions, this application provides a data processing method, apparatus, storage medium, and device. Compared with existing technologies, this application utilizes the similarities and differences between adjacent frames to extract data from the parts of the current frame image where similarity differs, while retaining data within the range of human visual attention. Transmitting the data from the parts of the current frame image where similarity differs, as well as the data within the range of human visual attention, to the client via the network reduces the amount of data transmitted over the network. Furthermore, because the human eye perceives differences between areas of focus and areas of non-focus, the differences in the image within the area of ​​focus are more pronounced. Based on this characteristic, this application retains data within the range of human visual attention, minimizing the risk of a decline in perceived image quality due to factors such as stitching quality. This reduces data volume while ensuring a good viewing experience for the user.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This illustration shows a scenario example provided by an embodiment of this application; Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application is shown; Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application is shown; Figure 4 This illustration shows a scenario example provided by an embodiment of this application; Figure 5 This illustration shows a scenario example provided by an embodiment of this application; Figure 6 This illustration shows a flowchart example provided in an embodiment of this application; Figure 7 This illustration shows a scenario example provided by an embodiment of this application; Figure 8 This illustration shows a scenario example provided by an embodiment of this application; Figure 9 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application is shown. Detailed Implementation

[0021] To facilitate the explanation of the embodiments of this application, some application scenarios related to the embodiments of this application are described below by way of example.

[0022] Scenario 1: In cloud gaming, the cloud server needs to render the game screen in real time and then convert the game screen into frame-by-frame image data (frame image stream). This frame image data is transmitted to the client via the network, and the client ultimately plays the game screen composed of these frame image data. Because the amount of frame image data obtained from rendering some large-scale games or high-definition games is large, the requirements for network transmission quality are high. If the network transmission quality does not meet the requirements, it will lead to a decrease in the clarity of the game screen played on the client, or cause problems such as frame drops and stuttering, affecting the user experience.

[0023] Scenario 2: In the metaverse and virtual social scenarios, real-time rendering via cloud servers is required to enable synchronous interaction among multiple users. If the amount of data rendered in real-time is large and the network transmission quality is insufficient, delays in the client's playback will occur, failing to meet the requirements for synchronous interaction among multiple users and impacting the user experience.

[0024] Existing technologies reduce data transmission by transmitting portions of the current frame that differ from the previous frame to the client. However, this method involves selecting portions of the entire frame that differ from the previous frame and transmitting them to the client, where they are then stitched, replaced, or smoothed out. These operations inevitably affect the quality of the smoothed-out portion, which cannot perfectly match the quality of the rendered image. This results in a noticeable decrease in the overall quality of the generated frame. If this degraded area falls within the area of ​​visual focus, the user will clearly perceive the change in image quality, negatively impacting their viewing experience.

[0025] This application provides a data processing method applicable to the aforementioned scenarios. This application utilizes the similarity and difference between adjacent frames to extract data from the portion of the current frame image where similarity differs, while retaining data within the range of human visual attention. The data from the portion with similarity differences, along with the data within the range of human visual attention, are transmitted to the client via a network. Since only a portion of the frame image data is transmitted over the network, the amount of data transmitted is reduced compared to transmitting the full amount of data, allowing even poor network conditions to meet data transmission requirements, thereby ensuring playback quality on the client side.

[0026] Furthermore, since users of cloud gaming or virtual social interaction tend to focus their attention on a specific area of ​​the screen (the area of ​​focus), this embodiment transmits the data of the area of ​​focus and areas with differences in similarity to the client. This avoids the problem of the data of the area of ​​focus being replaced or partially spliced, which would cause users to perceive a decrease in screen quality, thus ensuring the user's viewing experience.

[0027] The embodiments of this application can be applied to real-time streaming systems, such as... Figure 1 As shown, the real-time streaming system includes a cloud rendering server, a streaming module, and a client.

[0028] The client and the streaming module establish a real-time communication (RTC) connection, which can ensure that the latency during data transmission is controlled within an acceptable range. This is suitable for scenarios with high latency requirements, such as cloud gaming and virtual social networking.

[0029] The client can use devices such as cameras to collect the user's visual data and generate the region of interest data based on the user's visual data, and transmit the region of interest data to the streaming module via the network.

[0030] In addition, after receiving a portion of the current frame image data sent by the streaming module, the client can merge it with the pre-cached full frame image to obtain the data of the entire current frame image and play it.

[0031] The cloud rendering server and the streaming module can be connected via a communication network. The cloud rendering server uses its high-performance computing resources to render images in real time and transmits the corresponding frame image data to the streaming module via the network.

[0032] The streaming module can identify and extract the data of the first region corresponding to the region of interest data sent by the client. Furthermore, it can perform similarity calculations and other related logical steps to identify and extract the data of the second region (the part where the current frame image differs in similarity from adjacent frames). Next, the streaming module encodes the data of the first and second regions of the current frame image and transmits them to the client.

[0033] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0034] This embodiment provides a data processing method, such as Figure 2 As shown, the method includes: S101. Obtain the data of the area of ​​interest sent by the client.

[0035] The attention area data is generated based on the range of the terminal device screen that the human eye focuses on.

[0036] For example, region of interest data can be obtained by combining human eye focus areas with image acquisition devices and computer vision technology. The human eye focus area refers to the area in a visual scene that the human eye naturally focuses on or prioritizes processing.

[0037] In some examples, the client can be deployed on a terminal device. The client uses hardware such as a camera and computer vision technology to obtain the coordinates of the center point of the screen being focused on by the human eye, as well as the spatial information between the human eye and the screen (distance, height, deflection angle, etc.). Then, based on the center point coordinates, the spatial information between the human eye and the screen, and the range of the human eye's clear field of view, the radius of the region of interest is calculated. Finally, the region of interest data is obtained based on the center point coordinates and the radius of the region of interest.

[0038] S102. Based on the data of the region of interest, extract the data of the first region of the current frame image.

[0039] The data for the first region includes pixel data of pixels within the coordinate range of the first region.

[0040] In some examples, a mapping relationship can be established between the screen coordinate system and the current frame image coordinate system, mapping the center point coordinates of the region of interest data to the current frame image to obtain the origin of the first region's coordinates. Next, based on the origin of the first region's coordinates and the radius of the region of interest, the coordinate range of the first region is determined. Finally, pixel data located within the coordinate range of the first region is extracted.

[0041] In some examples, pixel data of the current frame image can be extracted using function methods. For instance, to extract pixel data of a specific region, you first need to determine the coordinate range of a first region in the frame image, and then filter out the pixel data within the circle using a mask.

[0042] Alternatively, the first region can also be a rectangular region. The center point of the rectangular region can be determined based on the origin of the coordinate system, and the side length of the rectangular region is obtained by multiplying the radius of the region of interest by 2. This application embodiment only uses a circular region as an example for illustration and does not impose any limitation on the shape of the first region. S103. Based on the pixel similarity between the current frame image and the adjacent frame images, extract the data of the second region of the current frame image.

[0043] Among them, the second region of the current frame image has a similarity difference with the second reference region of the adjacent frame image, and the coordinate range of the second region of the current frame image and the second reference region of the adjacent frame image is the same.

[0044] In some examples, an adjacent frame image can be the previous frame image of the current frame image. For instance, taking a frame image stream containing images i-2, i-3, and i-4 as an example, when extracting data from the second region of image i-3, pixel similarity is calculated between images i-2 and i-3. When extracting data from the second region of image i-4, similarity is calculated between images i-3 and i-4, and so on.

[0045] In some examples, a full frame image can be stored in a preset storage unit as the previous frame image of the current frame image. Pixel similarity calculations can be performed between the previous frame image and the current frame image. After part or all of the data of the current frame image is transmitted to the client, all the data of the previous frame image is overwritten in the preset storage unit with all the data of the current frame image.

[0046] In some examples, the pixel positions of the second region of the current frame image can be mapped to the coordinate space of the adjacent frame image through pixel position mapping, thus obtaining the second reference region of the adjacent frame image.

[0047] In some examples, pixel similarity can be obtained based on color space distance (e.g., Euclidean distance, Manhattan distance, etc.), color perception similarity (e.g., CIELab space distance, hue-saturation-brightness similarity, etc.), grayscale value similarity (e.g., absolute difference, cosine similarity, etc.), statistical or probabilistic methods (e.g., structural similarity index, normalized cross-correlation, etc.).

[0048] S104. Send the data of the first region and the second region of the current frame image to the client.

[0049] The data from the first region and the data from the second region are used to combine the full frame images periodically sent to the client to obtain the current frame image.

[0050] In some examples, the full frame image is used as a reference for the client to synthesize the data of the first region and the data of the second region of the current frame image obtained within the current image update cycle to obtain the current frame image.

[0051] In some examples, the client can determine the coordinate range of the missing portion of the current frame image based on data from the first and second regions of the current frame image. Then, the missing portion of the current frame image is filled in using the full frame image.

[0052] The data streaming method provided in the embodiments of this application will now be described in detail.

[0053] In some examples of this application, such as Figure 3 As shown, some specific implementation methods are provided for extracting data of the second region of the current frame image based on the pixel similarity between the current frame image and adjacent frame images, including: S1031. Divide the non-first region of the current frame image according to the preset rules to obtain multiple candidate regions.

[0054] In this context, the non-first region refers to any region other than the first region of the current frame image, which is equivalent to the region outside the range of human visual attention. The first region and the non-first region do not overlap, and the data from both regions constitute all the data of the current frame image. Candidate regions belong to the non-first region and are used to calculate pixel similarity with reference candidate regions in adjacent frames. The sum of the areas of multiple candidate regions can be less than or equal to the area of ​​the non-first region.

[0055] In some examples, the preset rule could be to divide the non-first region into equal areas based on a preset number of candidate regions, resulting in a preset number of candidate regions with the same area. Alternatively, it could be to divide the non-first region into a preset number of candidate regions and the shape of the first region, resulting in a preset number of candidate regions with the same shape as the first region.

[0056] Specifically, embodiments of this application provide several implementation methods for dividing the non-first region of the current frame image according to preset rules to obtain multiple candidate regions, including: Using the origin of the first region as the center, the radius of the candidate region is obtained by using the radius of the first region and a preset step size. The non-first region of the current frame image is divided to obtain multiple non-overlapping candidate regions. The radii of adjacent candidate regions differ by a preset step size.

[0057] In some examples, such as Figure 4As shown, the coordinate origin of the first region ( , Using a circle centered at (r) and with the radius of the first region and the preset step size (nγ) as the radius of the candidate region, the non-first region of the current frame image is divided. This results in non-overlapping regions A, B, and C. Here, γ is the length of the preset step size, and n is the number of steps within the preset step size. n can be obtained using the following equation: Formula 1 The candidate regions obtained in this embodiment are related to the first region, that is, to the range of human visual attention. Since human visual attention decreases from the visual center to the edge, dividing the candidate regions according to the range of human visual attention from near to far conforms to the distribution of human visual attention. Furthermore, it defines the basic range for subsequently extracting data from the second region of the current frame image.

[0058] S1032. Extract the pixels in the candidate region and calculate the pixel similarity with the pixels in the reference candidate region of the adjacent frame image to obtain the first similarity corresponding to multiple candidate regions.

[0059] Among them, the reference candidate region corresponds one-to-one with the candidate region, and the coordinate range is the same.

[0060] In some examples, the first similarity represents the overall pixel similarity between the candidate region and the reference candidate region.

[0061] In some examples, pixel similarity can be calculated by selecting all pixels of the candidate region and all pixels of the reference candidate region to obtain the first similarity of the candidate region.

[0062] In some examples, multiple sub-regions can be sampled from the candidate region. Based on the pixel range of these sub-regions, multiple reference sub-regions are selected from the reference candidate region. Then, pixels of the sub-regions and reference sub-regions are extracted and pixel similarity is calculated to obtain the first similarity of the candidate region.

[0063] Specifically, embodiments of this application provide specific implementation methods for extracting pixels within a candidate region and calculating pixel similarity with pixels in a reference candidate region of an adjacent frame image to obtain a first similarity corresponding to multiple candidate regions, including the following steps: Step 1: Sample the candidate region to obtain multiple sub-regions.

[0064] In some examples, the sub-region can be a regular shape such as a circle or rectangle, which makes it easier to obtain pixel coordinates.

[0065] Taking a circular sub-region as an example, in some examples, the radius of the sub-region is represented by r1, and the value of r1 ranges from... Based on the rules that sub-regions do not overlap and their range does not exceed that of the candidate region, the maximum number of sub-regions that can be sampled from the candidate region is obtained. This number can be represented by d. max express.

[0066] Based on the above Figure 4 The example shown illustrates how to divide a non-first region of the current frame image into multiple candidate regions. Sampling is performed on region A, which is the candidate region closest to the first region, to obtain the maximum number of samples Ad belonging to the sub-regions of region A. max Then, based on the maximum number of samples d in the sub-region max Set the maximum number of samples d for other areas (area B, area C). max They are Bd respectively. max and Cd max After obtaining the maximum number of samples for each candidate region's corresponding sub-region, sampling is performed based on this maximum number of samples to obtain multiple sub-regions for each candidate region. For example, such as... Figure 5 As shown, sampling of region A yields sub-regions A-d1, A-d2, and A-d3.

[0067] In addition, the sub-regions corresponding to each candidate region can be sampled based on the actual area of ​​each candidate region and the actual area of ​​the sub-region.

[0068] Step 2: Extract the pixels of multiple sub-regions included in the candidate region and calculate the pixel similarity with the pixels of the reference sub-region of the previous frame image to obtain the second similarity of multiple sub-regions; the reference sub-region and the sub-region correspond one-to-one and have the same coordinate range.

[0069] In some examples, the pixel similarity calculation method provided in the above examples can be used to obtain the second similarity of each sub-region.

[0070] Step 3: Obtain the first similarity of the candidate region based on the average of the second similarities of multiple sub-regions.

[0071] In some examples, as described above Figure 5 For example, the second similarity scores of the sub-regions A-d1, A-d2, and A-d3 included in region A are summed and averaged to obtain the first similarity score of region A. For instance, the second similarity score of sub-region A-d1 is sim. d1 The second similarity of the A-d2 subregion is sim d2 The second similarity of the A-d3 subregion is sim d3 The first similarity corresponding to region A The following equation (2) can be used as a reference: Formula 2 S1033. Based on the pixel distance between the reference pixels of the candidate region and the reference pixels of the first region, select candidate regions that meet the similarity conditions one by one from small to large until all candidate regions that meet the similarity conditions are selected, or there are candidate regions that do not meet the similarity conditions.

[0072] In some examples, the similarity condition is used to distinguish the parts of the current frame image that differ from the adjacent frame images. The regions that meet the similarity condition are those where the current frame image differs from the adjacent frame images or where the difference is significant, while the regions that do not meet the similarity condition are those where the current frame image does not differ from the adjacent frame images or where the difference is minor.

[0073] In some examples, the reference pixel of the first region can be the center point of the first region, and the reference pixel of the candidate region can be a pixel at a preset position, which can be the pixel corresponding to the maximum radius of the candidate region.

[0074] Specifically, this application embodiment also provides a specific implementation method for selecting candidate regions that meet the similarity condition one by one from small to large based on the pixel distance between the candidate region and the first region, including the following steps: Step 1: Obtain the first fusion value based on the first similarity of the candidate regions and the weight value corresponding to the candidate regions.

[0075] In some examples, the weight values ​​corresponding to the candidate regions can be set according to the characteristics of human visual attention distribution.

[0076] In some examples provided in this application, the weight value corresponding to the candidate region is negatively correlated with the pixel distance between the candidate region reference pixel and the first region reference pixel.

[0077] In this context, negative correlation indicates that the larger the pixel distance between the candidate region reference pixel and the first region reference pixel, the smaller its corresponding weight value.

[0078] In some examples, after obtaining the weight values ​​corresponding to the candidate regions, the first fusion value can be obtained based on the first similarity and the corresponding weight values ​​of the candidate regions.

[0079] Step 2: Obtain the overall similarity based on the first fusion value corresponding to the unselected candidate regions.

[0080] Among them, the unselected candidate regions represent candidate regions for which the process of extracting data has not yet begun.

[0081] In some examples, the overall similarity can be obtained by averaging the first fusion values ​​corresponding to the unselected candidate regions. If there is only one unselected candidate region, then the first fusion value of that candidate region is the overall similarity.

[0082] For example, the overall similarity can be obtained using the following Equation 3. : Formula 3 Step 3: If the overall similarity is less than the preset similarity threshold, then select the candidate region that is closest to the pixel distance of the first region from the unselected candidate regions.

[0083] Step 4: Return to the previous step and obtain the overall similarity based on the first fusion value corresponding to the unselected candidate regions, until there are no unselected candidate regions, or the overall similarity is greater than the preset similarity threshold.

[0084] In some examples, such as Figure 6 As shown, the logical execution steps for selecting candidate regions that meet the similarity criteria in the above example are as follows: S601. Obtain the overall similarity based on the first fusion value corresponding to the unselected candidate regions.

[0085] S602. Determine whether the overall similarity is less than the preset similarity threshold. If yes, proceed to S603; otherwise, proceed to S604.

[0086] S603. Select the candidate region that is closest to the pixels of the first region from the unselected candidate regions.

[0087] S604. Determine if there are any unselected candidate regions. If yes, execute S601; otherwise, execute S605.

[0088] S605. Obtain all selected candidate regions and stop execution.

[0089] By following the steps described above, the regions from which data needs to be extracted can be determined sequentially, from areas of high visual attention to areas of low attention, based on the distribution characteristics of human visual attention. This process continues until it is determined that data from all regions of the current frame image needs to be transmitted to the client, or that data from only a portion of the current frame image needs to be transmitted to the client.

[0090] S1034. Extract the data of candidate regions that meet the similarity criteria.

[0091] Candidate regions that meet the similarity criteria are classified as the second region.

[0092] In some examples, the data extraction method described above can be used to extract data by means of a mask after obtaining the pixel coordinates of candidate regions that meet the similarity conditions.

[0093] In some examples of this application, the specific implementation of obtaining the current frame image and determining whether the current frame image has been fully transmitted to the client before processing the current frame image includes the following steps: Step 1: Obtain the current frame image.

[0094] Step 2: If the current frame image is a keyframe image, or if the full frame image has not been sent to the client within the current image update cycle, transmit the data of all regions of the current frame image to the client.

[0095] In this context, a keyframe image refers to a frame image used to define change nodes in a frame image data stream. Keyframe images record crucial image information in the sequence, such as the position, shape, color, and transparency of objects.

[0096] In some examples, at least one full frame image can be sent to the client during an image update cycle. This full frame image is used to combine data from the first and second regions of the current frame image to generate a complete current frame image.

[0097] If a full frame image has already been sent to the client during an image update cycle, but the current frame image is a keyframe image, then the current frame image is sent to the client. Upon receiving the current frame image, the client overwrites the full frame image already acquired in the current image update cycle.

[0098] Specifically, based on the above embodiments of this application, this application also provides a specific implementation method, including: Step 1: At the beginning of the streaming phase, the client requests camera access from the terminal device to collect spatial information about the human eye and the screen. Based on this spatial information and computer vision technology, the coordinates of the center point of the screen that the human eye is focusing on are obtained. Then, based on the center point coordinates and the clear field of view of the human eye... The radius of the region of interest is calculated. Then, the collected center point coordinates and the radius of the region of interest (region of interest data) are transmitted to the streaming module.

[0099] Step 2: The streaming module presets an image update cycle and sends at least one full-frame image to the client within one image update cycle. If the streaming module acquires a keyframe image within one image update cycle, it transmits the keyframe as a full-frame image to the client. If no keyframe image is acquired within one cycle, the acquired current frame image is transmitted as a full-frame image to the client.

[0100] Step 3: For the non-interested region (non-first region), divide it into multiple annular regions (candidate regions) according to a preset step size. Sample a certain number of sub-regions for each candidate region and calculate their similarity to obtain the second similarity of each sub-region. Then, calculate the average of the second similarities of each sub-region as the first similarity of the corresponding candidate region. After obtaining the first similarity of each candidate region, recursively calculate the overall similarity of the non-interested region according to the weights, and compare the overall similarity with the preset similarity threshold. The comparison is performed. If the overall similarity is less than a preset similarity threshold, the candidate region with the closest pixel distance to the first region is selected from the unselected candidate regions. This process is repeated until there is a case where the overall similarity is greater than the preset similarity threshold, at which point the data of the unselected candidate regions is cropped. The selected candidate regions (the difference portion) and the data of the first region (the portion of human eye focus) are retained and transmitted to the client. If, after calculation, there is no case where the overall similarity is greater than the preset similarity threshold, the entire current frame image is transmitted to the client.

[0101] like Figure 7 As shown, when the overall similarity between regions B and C is greater than a preset similarity threshold, the coordinates of the center of the region of interest are used to determine the similarity. , ), and crop the B and C regions of the current frame image by cropping the radius of the cropping circle.

[0102] Step 4: After receiving the current frame image, the client parses the missing parts of the current frame data from the metadata, then fills in the missing parts based on the full frame images saved in the current image update cycle to obtain the complete current frame image, and then plays the complete current frame image. For example... Figure 8 As shown, the current frame image is obtained by adding a portion of the current frame image (the area of ​​human eye focus and the area of ​​difference) to the full frame image.

[0103] Furthermore, this embodiment provides a data processing device, such as... Figure 9 As shown, the device includes: an acquisition module 110, an extraction module 120, a calculation module 130, and a transmission module 140.

[0104] The acquisition module 110 is configured to acquire attention area data sent by the client; the attention area data is generated based on the range of the human eye's focus on the terminal device screen; Extraction module 120 is configured to extract data of the first region of the current frame image based on the region of interest data; The calculation module 130 is configured to extract data of a second region of the current frame image based on the pixel similarity between the current frame image and the adjacent frame images; the second region of the current frame image has a similarity difference with the second region of the adjacent frame images, and the coordinate range of the second region of the current frame image and the second region of the adjacent frame images is the same; The sending module 140 is configured to send data of the first region and the second region of the current frame image to the client; the data of the first region and the data of the second region are used to combine with the full frame images periodically sent to the client to obtain the complete current frame image.

[0105] In some examples provided in this application, the calculation module 130 is specifically configured to divide the non-first region of the current frame image according to a preset rule to obtain multiple candidate regions; Pixels within a candidate region are extracted and their similarity to pixels in a reference candidate region of an adjacent frame image is calculated to obtain the first similarity for each candidate region. The reference candidate region corresponds one-to-one with the candidate region and has the same coordinate range. Based on the pixel distance between the reference pixels of the candidate region and the reference pixels of the first region, candidate regions that meet the similarity conditions are selected one by one from small to large until all candidate regions that meet the similarity conditions are selected, or there are candidate regions that do not meet the similarity conditions. Extract candidate regions that meet the similarity criteria, and assign these candidate regions to the second region.

[0106] In some examples provided in this application, the calculation module 130 is further specifically configured to divide the non-first region of the current frame image into multiple non-overlapping candidate regions by taking the origin of the first region as the center, the radius of the first region and a preset step size as the radius of the candidate region; the radii of adjacent candidate regions differ by a preset step size.

[0107] In some examples provided in this application, the calculation module 130 is further specifically configured to extract pixels within a candidate region and perform pixel similarity calculations with pixels in a reference candidate region of an adjacent frame image to obtain first similarities corresponding to multiple candidate regions, including: The candidate region is sampled to obtain multiple sub-regions; The pixels of multiple sub-regions included in the candidate region are extracted and their pixel similarity is calculated with the pixels of the reference sub-region of the previous frame image to obtain the second similarity of multiple sub-regions; the reference sub-region corresponds one-to-one with the sub-regions and has the same coordinate range; The first similarity of the candidate region is obtained by averaging the second similarities of multiple sub-regions.

[0108] In some examples provided in this application, the calculation module 130 is further configured to obtain a first fusion value based on the first similarity of the candidate regions and the weight value corresponding to the candidate regions; the weight value corresponding to the candidate regions is determined based on the pixel distance between the candidate regions and the first regions; The overall similarity is obtained based on the first fusion value corresponding to the unselected candidate regions; If the overall similarity is less than the preset similarity threshold, then the candidate region that is closest to the pixel distance of the first region is selected from the unselected candidate regions. Return to the step of obtaining the overall similarity based on the first fusion value corresponding to the unselected candidate regions, until there are no unselected candidate regions, or the overall similarity is greater than the preset similarity threshold.

[0109] In some examples provided in this application, the device further includes an identification module configured to acquire the current frame image; and to transmit the data of all regions of the current frame image to the client if the current frame image is a keyframe image or if the full frame image has not been sent to the client within the current image update cycle.

[0110] It should be noted that other corresponding descriptions of the functional units involved in the data processing apparatus provided in this embodiment can be found in the description of the data processing method in the above embodiments, and will not be repeated here.

[0111] Based on the data processing method shown in the above embodiments, this embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method shown in the above embodiments.

[0112] Based on the methods shown in the above embodiments, this embodiment also provides a computer program product on which a computer program is stored, and when the computer program product is executed by a processor, it implements the methods shown in the above embodiments.

[0113] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0114] Based on the method shown in the above embodiments, and Figure 9To achieve the above objectives, the present application also provides an electronic device, such as a terminal device, in the virtual device embodiment shown. The electronic device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the method shown in the above embodiment.

[0115] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0116] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0117] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented through hardware. Compared with current related technologies, this application implements differentiated comparison of frame images on the server side and retains data within the range of human eye focus. Transmitting data with similarity differences and data within the range of human eye focus to the client via the network can reduce the amount of data transmitted over the network. Furthermore, due to the difference between clear and blurred fields of view, the human eye perceives significant differences in image differences within the clear field of view. This application retains data within the range of human eye focus, which can minimize the decrease in perceived image quality caused by factors such as stitching quality. While reducing data volume, it ensures the user's viewing experience.

[0119] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method, characterized in that, include: Retrieve data on the area of ​​interest sent by the client; The attention area data is generated based on the range of the human eye's focus on the terminal device screen. Based on the region of interest data, extract the data of the first region of the current frame image; Based on the pixel similarity between the current frame image and the adjacent frame images, data of the second region of the current frame image is extracted; the second region of the current frame image has a similarity difference with the second reference region of the adjacent frame images, and the coordinate range of the second region of the current frame image and the second reference region of the adjacent frame images is the same; The data of the first region and the data of the second region of the current frame image are sent to the client; the data of the first region and the data of the second region are used to combine with the full frame images periodically sent to the client to obtain the complete current frame image.

2. The method according to claim 1, characterized in that, Based on the pixel similarity between the current frame image and adjacent frame images, data of the second region of the current frame image is extracted, including: The non-first region of the current frame image is divided according to a preset rule to obtain multiple candidate regions; Pixels within the candidate region are extracted and their similarity is calculated with pixels within the reference candidate region of the adjacent frame image to obtain the first similarity corresponding to each of the multiple candidate regions; the reference candidate region corresponds one-to-one with the candidate region and has the same coordinate range. Based on the pixel distance between the reference pixel of the candidate region and the reference pixel of the first region, candidate regions that meet the similarity condition are selected one by one from small to large until all candidate regions that meet the similarity condition are selected, or there are candidate regions that do not meet the similarity condition. Data of candidate regions that meet the similarity criteria are extracted, and the candidate regions that meet the similarity criteria belong to the second region.

3. The method according to claim 2, characterized in that, The non-first region of the current frame image is divided according to a preset rule to obtain multiple candidate regions, including: Using the origin of the first region as the center, the radius of the candidate region is obtained by using the radius of the first region and a preset step size. The non-first region of the current frame image is divided to obtain multiple non-overlapping candidate regions. The preset step size includes the length of the preset step size and the number of preset step sizes. The radii of adjacent candidate regions differ by the length of the preset step size.

4. The method according to claim 2, characterized in that, The step of extracting pixels within the candidate region and calculating pixel similarity with pixels in the reference candidate region of the adjacent frame image to obtain a first similarity corresponding to multiple candidate regions includes: The candidate region is sampled to obtain multiple sub-regions; The pixels of multiple sub-regions included in the candidate region are extracted and their pixel similarity is calculated with the pixels of the reference sub-region of the previous frame image to obtain the second similarity corresponding to each sub-region; the reference sub-region corresponds one-to-one with the sub-region and has the same coordinate range; The first similarity of the candidate region is obtained by averaging the second similarities of multiple sub-regions.

5. The method according to claim 4, characterized in that, The step of selecting candidate regions that meet the similarity condition one by one from small to large based on the pixel distance between the candidate regions and the first region includes: A first fusion value is obtained based on the first similarity of the candidate regions and the weight value corresponding to the candidate regions; The overall similarity is obtained based on the first fusion value corresponding to the unselected candidate regions; If the overall similarity is less than a preset similarity threshold, then the candidate region with the closest pixel distance to the first region is selected from among the multiple unselected candidate regions. Return to the step of obtaining the overall similarity based on the first fusion value corresponding to the unselected candidate regions, until there are no unselected candidate regions, or the overall similarity is greater than a preset similarity threshold.

6. The method according to claim 5, characterized in that, The weight value corresponding to the candidate region is negatively correlated with the pixel distance between the reference pixel of the candidate region and the reference pixel of the first region.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Obtain the current frame image; If the current frame image is a keyframe image, or if the full frame image has not been sent to the client within the current image update cycle, the data of all regions of the current frame image will be transmitted to the client.

8. A data processing apparatus, characterized in that, include: The acquisition module is configured to acquire data about the area of ​​interest sent by the client. The attention area data is generated based on the range of the human eye's focus on the terminal device screen. The extraction module extracts data from the first region of the current frame image based on the region of interest data. The calculation module is configured to extract data of a second region of the current frame image based on the pixel similarity between the current frame image and the adjacent frame images; the second region of the current frame image has a similarity difference with the second region of the adjacent frame images, and the coordinate range of the second region of the current frame image and the second region of the adjacent frame images is the same; The sending module is configured to send data of the first region and the second region of the current frame image to the client; the data of the first region and the data of the second region are used to combine with the full frame images periodically sent to the client to obtain the complete current frame image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

11. A computer program product having a computer program stored thereon, characterized in that, When the computer program product is executed by a processor, it implements the method of any one of claims 1 to 7.

Images