Interaction-based encoding method, encoding device, and readable storage medium

Abstract

The application discloses an interactive-based coding method, a coding device and a readable storage medium, wherein the coding method comprises the following steps: receiving user interaction information of a current frame to be coded and determining a prediction change vector of the current frame to be coded according to the interaction information; searching a history data linked list according to pixels of the current frame to be coded and the prediction change vector to perform motion vector search, and obtaining an initial matching block through matching; expanding a matching area according to the initial matching block to obtain a maximum matching area; performing area deduplication on adjacent frames of the current frame to be coded according to the maximum matching area to obtain a non-repeated area of the adjacent frames; encoding the non-repeated area of the adjacent frames, and storing the non-repeated area and a vector identifier associated with the non-repeated area into the history data linked list. The embodiment of the application combines operation information of an interactive device, improves the prediction performance of area deduplication, shortens the coding time consumption, and improves the fluency of a real-time interactive scene.

Description

Technical Field

[0001] This invention relates to the field of interactive real-time encoding technology, and in particular to an interactive encoding method, encoding device, and readable storage medium. Background Technology

[0002] In applications that rely on real-time interactive screen content encoding, such as cloud desktop applications and virtual reality technologies, users frequently interact using devices like keyboards, mice, touchscreens, and gamepads, causing the screen to move or scroll horizontally, vertically, or in other unpredictable directions. In these scenarios, to promptly encode and push the changing screen content to the other end for display, deduplication of the encoding area is typically employed to reduce the encoding area and shorten encoding time.

[0003] In related technologies, deduplication of the region to be encoded typically involves generating a hash value from the already encoded data and caching it in memory or a database. Subsequent updates generate a new hash value, and matching and matching are performed based on this hash value. However, hash-based solutions experience a significant increase in CPU resource consumption and computation time when dealing with high resolutions, impacting the real-time performance of the encoding. Summary of the Invention

[0004] This invention provides an interactive encoding method, encoding device, and readable storage medium that can encode based on the input of interactive devices, thereby improving encoding speed and interactive fluency.

[0005] In a first aspect, embodiments of the present invention provide an interactive encoding method, comprising:

[0006] Receive user interaction information on the current frame to be encoded and determine the predicted change vector of the current frame to be encoded based on the interaction information, wherein the interaction information is used to characterize the user's operation on the screen through the interactive device to move and / or scroll at least a portion of the pixels in the screen in at least one direction.

[0007] Based on the pixels of the current frame to be encoded and the predicted change vector, the historical data list is searched to perform motion vector search, and an initial matching block and a precise motion vector are obtained. The historical data list stores non-repeating regions before and after historical image changes and vector identifiers associated with the non-repeating regions. The vector identifiers are used to characterize the change vectors before and after historical image changes.

[0008] The matching region is expanded based on the initial matching block and the precise motion vector to obtain the maximum matching region;

[0009] Based on the maximum matching region, perform region deduplication on the current frame to be encoded to obtain the non-repeating region of the current frame to be encoded.

[0010] The non-repeating regions of the current frame to be encoded are encoded, and the non-repeating regions and the vector identifiers associated with the non-repeating regions are stored in the historical data linked list.

[0011] In a second aspect, embodiments of the present invention provide an encoding apparatus, including at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the encoding method as described in the first aspect.

[0012] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the encoding method as described in the first aspect.

[0013] The interactive encoding method, encoding device, and readable storage medium provided in this invention have at least the following beneficial effects: when a user operates an interactive device, causing the screen to move or scroll, the approximate motion mode of the current frame to be encoded can be determined based on the operation of the interactive device, thereby determining the predicted change vector of the current frame to be encoded. Based on the predicted change vector, motion vector search is performed to determine the repeating and non-repeating regions, significantly improving the deduplication efficiency of the encoding region in interactive scenarios where the screen moves or scrolls, and achieving the goals of improving encoding speed and accuracy, and improving the smoothness of use in interactive scenarios. Attached Figure Description

[0014] Figure 1 This is an overall flowchart of an encoding method provided in one embodiment of the present invention;

[0015] Figure 2 This is a flowchart of an embodiment of the present invention for determining whether interactive information triggers screen movement or scrolling;

[0016] Figure 3 This is a flowchart of calculating the predicted change vector provided in one embodiment of the present invention;

[0017] Figure 4 This is a flowchart of obtaining accurate motion vectors through motion vector search according to an embodiment of the present invention;

[0018] Figure 5 This is a flowchart illustrating how an extended region yields the maximum matching region, provided in one embodiment of the present invention.

[0019] Figure 6 This is a flowchart of recording non-repeating regions into a historical data linked list according to an embodiment of the present invention;

[0020] Figure 7 This is a schematic diagram of the structure of an encoding device provided in one embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] In cloud desktop scenarios based on real-time interactive screen content encoding, and in virtual reality applications with extremely high real-time requirements, users frequently interact with the screen using keyboards, mice, touchscreens, controllers, and VR headsets, performing rapid dragging or scrolling operations in horizontal, vertical, or other non-fixed directions. In these scenarios, to promptly encode and push the changing screen content to the receiving end, it's necessary to reduce the area to be encoded and shorten encoding time by deduplicating the screen to be encoded.

[0023] Common deduplication methods in related technologies involve generating hash values ​​for encoded data and caching these hash values ​​in memory or a database. Subsequent updates to the displayed content generate new hash values, which are then matched against the cached hash values. A match prevents duplicate encoding. However, hash-based region matching and deduplication schemes experience a significant increase in CPU resources and processing time for hash value calculations when the screen resolution exceeds a certain size, impacting real-time performance.

[0024] Based on this, embodiments of the present invention provide an interactive encoding method, encoding device, and readable storage medium, which combines the input of the interactive device to identify the movement and scrolling of the screen during the interaction, and further predicts the motion vector, thereby achieving accurate displacement calculation, realizing rapid identification of repeated areas, accelerating the encoding speed and reducing the encoding time.

[0025] Reference Figure 1 The encoding method provided in this embodiment of the invention includes, but is not limited to, the following steps S100 to S500.

[0026] Step S100: Receive user interaction information on the current frame to be encoded and determine the predicted change vector of the current frame to be encoded based on the interaction information. The interaction information is used to characterize the user's operation on the screen through the interactive device to move and / or scroll at least a portion of the pixels in the screen in at least one direction.

[0027] Step S200: Based on the pixels of the current frame to be encoded and the predicted change vector, search the historical data list to perform motion vector search, and obtain the initial matching block and the precise motion vector. The historical data list stores the non-repeating regions before and after the changes in the historical image and the vector identifiers associated with the non-repeating regions. The vector identifiers are used to characterize the change vectors before and after the changes in the historical image.

[0028] Step S300: Expand the matching region based on the initial matching block and the precise motion vector to obtain the maximum matching region;

[0029] Step S400: Perform region deduplication on the current frame to be encoded based on the maximum matching region to obtain the non-repeating region of the current frame to be encoded;

[0030] Step S500: Encode the non-repeating region of the current frame to be encoded, and store the non-repeating region and the vector identifier associated with the non-repeating region into the historical data linked list.

[0031] The encoding method of this invention is applied to interactive systems, which include interactive devices through which users can input, causing changes to the screen within the interactive system. For example, in cloud desktop applications, the interactive system is an interaction between the user terminal and the cloud. The cloud receives user input, processes user data, and then sends the screen to the user terminal for display in real time, requiring the screen transmission time to be minimized. Similarly, in virtual reality technology applications, the interactive system is a complete VR device. Users can input through VR headsets or controllers, and the VR device provides corresponding screen feedback to the user based on the input, again requiring the screen transmission time to be minimized. Therefore, this invention provides a solution for fast encoding and improved user smoothness for this specific scenario of interactive systems. Specifically, firstly, the user's interaction information with the current frame to be encoded is determined based on the operation of the interactive device. Then, based on the interaction information, it is predicted whether the current frame to be encoded needs to scroll or move. If the current frame to be encoded needs to scroll or move, a predicted change vector is derived based on the interaction information. This interactive information is triggered by the user. The predicted change vector of the current frame to be encoded is predicted based on the user's actions. Users can interact with the screen using a keyboard, mouse, touchscreen, controller, VR headset, etc. These interactions are captured by the interactive system, which then calculates the predicted change vector. The predicted change vector consists of a predicted direction and a predicted displacement, representing the vector between two consecutive frames from the start point to the end point. It can be understood that this interactive information does not necessarily cause the entire current frame to be encoded to move or scroll; it can be a part of the current frame that moves or scrolls. For example, when a user scrolls the mouse wheel, part of the content on a webpage scrolls while the rest remains stationary. Similarly, when a user plays a game using a VR headset and looks up, the visual portion of the VR screen moves while the game UI remains stationary.

[0032] For the current frame to be encoded and the predicted change vector, within a certain pixel range, the historical data list is searched according to the pixels of the current frame to be encoded and the predicted change vector to perform motion vector search. The historical data list consists of non-repeating regions obtained from historical records and vector identifiers associated with the non-repeating regions. During the search, one or more records are locked from the historical data list based on the association relationship recorded by the predicted change vector, thereby matching the initial matching block and the precise motion vector.

[0033] Then, the initial matching block found after the above search is expanded into a matching region. The expansion method is determined based on the precise motion vector, resulting in the maximum matching region. This maximum matching region is then used for deduplication to obtain a non-repeating region. Subsequent encoding can then be performed, with repetitive regions identified. The encoded bitstream is then sent to the peer device, which identifies the repetitive regions based on the identifiers in the bitstream and decodes them to obtain the non-repeating regions.

[0034] Understandably, after dividing the region into repeating and non-repeating regions, the non-repeating regions need to be recorded in a historical data list for subsequent searching and deduplication of the data to be encoded during screen motion. This historical data list can be stored in memory, a database, or a reference frame; no restrictions are placed here.

[0035] In the case of an input device in a remote desktop scenario, before determining the interaction information, it is also necessary to determine whether the input of the current interaction device can trigger screen movement or scrolling. Specifically, refer to... Figure 2 The above step S100 also includes:

[0036] Step S110: Receive user operation information on the input device;

[0037] Step S120: When it is determined that the operation information triggers at least a portion of the pixels in the current frame to be encoded to move and / or scroll in at least one direction, the operation information is used as interaction information.

[0038] In remote desktop or cloud desktop scenarios, users typically use devices like mice and keyboards for interaction. However, mouse and keyboard inputs don't always trigger movement or scrolling of the frame to be encoded. For example, in a window where scrolling is restricted, holding down the left mouse button and moving the mouse a short distance to the right won't move the frame (the cursor doesn't move when it reaches the left edge of the screen). In this case, the interactive system doesn't need to respond to this input to execute the encoding method described above. When a user's interaction causes at least a portion of the pixels in the frame to be encoded to move and / or scroll in at least one direction, this user action is considered a valid interactive behavior that triggers the encoding method described above.

[0039] Reference Figure 3 The prediction change vector of the current frame to be encoded, as described above, can be determined through the following steps:

[0040] Step S130: Predict the motion direction and motion distance of the current frame to be encoded based on the interaction information;

[0041] Step S140: Determine the predicted change vector based on the direction and distance of motion.

[0042] Interactive information can represent a relatively complex movement trajectory, but the starting and ending points of the trajectory are definite. Therefore, between two consecutive frames, a precise motion vector can be determined based on the starting point in the previous frame and the ending point in the next frame. At this point, based on the predicted motion direction and distance from the interactive information, a rough predicted change vector can be directly calculated. It is understood that steps S130 and S140 above are applicable to both cloud desktop scenarios and virtual reality technology application scenarios. Although the input methods differ, the changes reflected on the screen can still be represented by vectors.

[0043] Reference Figure 4 The motion vector search process includes the following steps:

[0044] Step S210: Divide the current frame to be encoded into multiple blocks according to the pixel distribution pattern, and set the predicted change vector as the initial search vector;

[0045] Step S220: Select several blocks from the multiple blocks obtained by division as the starting point of motion vector search, and perform motion vector search within a certain range of the initial search vector to obtain the precise matching block and the precise motion vector.

[0046] Based on a block matching algorithm, blocks are matched within a certain pixel range to determine precise motion vectors. While related technologies typically use reference frames for block matching, this invention uses a historical data list for matching. As previously mentioned, the historical data list, formed by non-repeating regions and associated vector identifiers during the execution of this invention, allows for matching of the divided blocks within the historical data list. This fully utilizes the non-repeating regions in multiple consecutive frames for searching and deduplication. Compared to using reference frames, which typically only represent the changes between the current frame to be encoded and several nearby frames, using a stored historical data list effectively expands the reference region, providing search and deduplication assistance throughout the entire frame generation process.

[0047] The process of dividing the frame into several blocks described above can be based on the distribution pattern of the pixels in the current frame to be encoded. For example, for a fixed-resolution image, the current frame to be encoded can be divided into blocks of a fixed size grid without considering the content of the current frame to be encoded. Alternatively, it can be divided into multiple blocks of different sizes automatically based on the pixel colors of the current frame to be encoded. Or, it can be divided based on the hash values ​​of the pixels. Specifically, it includes the following steps:

[0048] Based on the pixels of the current frame to be encoded, the current frame to be encoded is divided according to the size of the encoding unit;

[0049] The content of the block is either the original pixel value or a hash value calculated based on the pixel value.

[0050] It is understood that the block size can be based on the size of the coding unit, or it can be a more refined division under the coding unit, or a division larger than the size of the coding unit, etc. There are no limitations here, and those skilled in the art can set the block size according to actual needs.

[0051] Reference Figure 5 Based on the initial matching block obtained after the above motion vector search, the matching region is expanded, including the following steps:

[0052] Step S310: Expand the initial matching block in the horizontal and vertical directions of the precise motion vector to determine the upper boundary, lower boundary, left boundary and right boundary, thus forming the matching boundary;

[0053] Step S320: On each horizontal row of the matching boundary, determine the left and right boundaries of the horizontal row;

[0054] Step S330: Determine the maximum matching region based on all horizontal rows whose matching region formed by the left and right boundaries is less than a preset threshold.

[0055] First, the maximum matching boundaries are found in the horizontal and vertical directions of the initial matching block. The horizontal and vertical distances are determined by the precise motion vector, which is actually a vector decomposition of the precise motion vector. Then, the left and right boundaries are determined in each horizontal row until the matching area in the row is less than a preset threshold. The maximum matching area is obtained after the matching is completed. The preset threshold is set according to the actual accuracy requirements, such as setting it to 50% of the maximum left and right boundaries.

[0056] After the deduplication process described above, duplicate and non-duplicate regions are obtained. After encoding the non-duplicate regions, they and related information need to be recorded in the historical data linked list, referring to... Figure 6 Specifically, it includes:

[0057] Step S510: Obtain the region size information, pixel information, region identifier, and precise motion vector associated with the non-repeating region;

[0058] Step S520: Store the region size information, pixel information, and region identifier of the non-repeating region after the latest record in the historical data list, and associate the latest record in the historical data list with the precise motion vector associated with the non-repeating region.

[0059] For example, when the current frame to be encoded and its adjacent frames are in a downward motion state, non-repeating regions can be associated with the latest record in the historical data list when added to the historical data list, and are marked as downward association; similarly, when the current frame to be encoded and its adjacent frames are in an upward motion state, non-repeating regions can be associated with the latest record in the historical data list when added to the historical data list, and are marked as upward association.

[0060] Finally, the encoding method can also be selected according to actual needs. For example, depending on the different real-time requirements, one of the following two encoding methods can be used:

[0061] When using shallow compression technology for encoding, the repetitive areas in the image are removed and the non-repetitive areas are encoded. The repetitive areas do not need to be encoded after being marked in the bitrate.

[0062] When using deep compression technology for encoding, repeating regions and vector identifiers associated with repeating regions are set in the encoder, while non-repeating regions are encoded using deep compression technology.

[0063] Shallow compression is suitable for scenarios with high real-time requirements, as the encoding and decoding process takes less time, but it consumes a larger bitrate. It is suitable for applications in virtual reality technology. Deep compression is suitable for scenarios with slightly lower real-time requirements, as the encoding and decoding process takes a slightly longer time, but it consumes a smaller bitrate. It is suitable for applications in cloud desktops.

[0064] Through the above steps, when the user operates the interactive device and the screen moves or scrolls, the approximate motion mode of the current frame to be encoded can be determined based on the operation of the interactive device. This allows for the determination of the predicted change vector of the current frame to be encoded. Based on the predicted change vector, motion vector search is performed to identify repeating and non-repeating regions. This significantly improves the deduplication efficiency of the encoded region in interactive scenarios where the screen moves or scrolls, achieving the goals of improving encoding speed and accuracy, and enhancing the smoothness of use in interactive scenarios.

[0065] The encoding method of the present invention will be illustrated below through two examples.

[0066] Example 1: Encoding methods applied in cloud desktop scenarios.

[0067] The implementation on the encoding side includes the following steps:

[0068] 1. Receive mouse button information, movement information, keyboard button information, gamepad button information, and joystick information, and identify whether the above input information is a scrolling or movement scenario;

[0069] 2. For the scrolling or moving scenarios identified above, predict the motion direction of the current frame to be encoded. The motion direction is not fixed up, down, left, right or certain angles. For example, predict the motion direction of the current frame to be encoded based on the movement trajectory of the mouse during mouse dragging. The motion trajectory can be used to calculate a rough motion displacement.

[0070] 3. Based on the identified motion direction and displacement, within a certain pixel range, a motion vector search is performed according to the original pixels of the blocks after the current frame to be encoded, or based on the hash values ​​generated from the pixels. During the search, a historical data list recorded during the motion process can be referenced. A better method is to lock one or more records from the linked lists of the historical data list based on the predicted change vector.

[0071] 4. Based on the initial matching block obtained from the above motion vector search, expand the matching area. First, find the largest matching boundary in the horizontal and vertical directions of the matching block. Then, determine the left and right boundaries of the matching in each horizontal row until the matching area in the row is smaller than the preset threshold. After the matching is completed, the maximum matching area and the precise motion vector are obtained.

[0072] 5. Based on the maximum matching region and precise motion vector determined after the above search expansion, remove duplicate regions or set motion regions, motion vectors, reference frame information, etc. to the encoder for fast compression during the encoding process;

[0073] 6. Non-repeating regions generated during scrolling and movement need to be recorded in memory, database, or reference frame for subsequent searching and deduplication of data to be encoded during motion. When recording regions, it is necessary to record their association with existing records, the size of the region, the pixel information of the region, and the ID information of the region.

[0074] The decoding process includes the following steps:

[0075] 1. Analyze the decoded bitstream. If the bitstream carries information about repeated regions, restore the region pixels based on the region's location, size, the ID of the historical record used, and the starting point. When the encoding end is deep compression, repeated regions can also be expressed through the use of reference frames and normal motion search. Therefore, the decoded bitstream does not necessarily contain information about repeated regions, and this step is unnecessary.

[0076] 2. Parse the decoded bitstream. If the bitstream carries information about newly added record regions, save the record according to its position, region size, and ID value transmitted by the encoder. When the encoder uses deep compression, the newly added region exists in the reference frame and is managed in the form of the reference frame. Therefore, the decoded bitstream does not necessarily contain information about newly added records, and this step is unnecessary.

[0077] Example 2: Encoding methods applied in VR scenarios.

[0078] The implementation on the encoding side includes the following steps:

[0079] 1. Receive motion vector information from the head-mounted device's sensors. The motion information from the head-mounted device can include movement, reversal, angular rotation, etc.

[0080] 2. For the motion vectors obtained from the head-mounted device, within a certain pixel range, a search for motion vectors is performed based on the original pixels of the blocks after the current frame to be encoded. During the search, a linked list of historical data recorded during the motion process can be referenced. A better method is to lock one or more records from the linked list of historical data based on the predicted change vectors.

[0081] 3. Based on the initial matching block obtained from the above motion vector search, expand the matching area. First, find the largest matching boundary in the horizontal and vertical directions of the matching block. Then, determine the left and right boundaries of the matching in each horizontal row until the matching area in the row is smaller than the preset threshold. After the matching is completed, the maximum matching area and the accurate motion vector are obtained.

[0082] 4. Based on the maximum matching region and precise motion vector determined after the above search expansion, remove duplicate regions or set motion regions, motion vectors, reference frame information, etc. for the encoder for fast compression during the encoding process;

[0083] 5. Non-repeating regions generated during scrolling and movement need to be recorded in memory, database, or reference frame for subsequent searching and deduplication of data to be encoded during motion. When recording regions, it is necessary to record their association with existing records, the size of the region, the pixel information of the region, and the ID information of the region.

[0084] The decoding process includes the following steps:

[0085] 1. Analyze the decoded bitstream. If the bitstream carries information about repeated regions, restore the region pixels based on the region's location, size, the ID of the historical record used, and the starting point. When the encoding end is deep compression, repeated regions can also be expressed through the use of reference frames and normal motion search. Therefore, the decoded bitstream does not necessarily contain information about repeated regions, and this step is unnecessary.

[0086] 2. Parse the decoded bitstream. If the bitstream carries information about newly added record regions, save the record according to its position, region size, and ID value transmitted by the encoder. When the encoder uses deep compression, the newly added region exists in the reference frame and is managed in the form of the reference frame. Therefore, the decoded bitstream does not necessarily contain information about newly added records, and this step is unnecessary.

[0087] This invention also provides an encoding apparatus, including at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the aforementioned encoding method.

[0088] Reference Figure 7 For example, the control processor 1001 and memory 1002 in the encoding device 1000 can be connected via a bus. The memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, the memory 1002 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1002 may optionally include memory remotely located relative to the control processor 1001, and these remote memories can be connected to the encoding device 1000 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0089] Those skilled in the art will understand that Figure 7 The device structure shown does not constitute a limitation on the encoding device 1000, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0090] This invention also provides a computer-readable storage medium storing computer-executable instructions that are executed by one or more control processors, for example, by... Figure 7 One of the control processors 1001 executes the above-described method, which can cause the one or more control processors to execute the encoding method in the above-described method embodiments, for example, to execute the above-described method. Figure 1 Method steps S100 to S500 Figure 2 Method steps S110 to S120 Figure 3 Method steps S130 to S140, Figure 4 Method steps S210 to S220, Figure 5 Method steps S310 to S330 and Figure 6Method steps S510 to S520.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. An interactive coding method, comprising: Receive user interaction information on the current frame to be encoded and determine the predicted change vector of the current frame to be encoded based on the interaction information, wherein the interaction information is used to characterize the user's operation on the screen through the interactive device to move and / or scroll at least a portion of the pixels in the screen in at least one direction. Based on the pixels of the current frame to be encoded and the predicted change vector, the historical data list is searched to perform motion vector search, and an initial matching block and a precise motion vector are obtained. The historical data list stores non-repeating regions before and after historical image changes and vector identifiers associated with the non-repeating regions. The vector identifiers are used to characterize the change vectors before and after historical image changes. The matching region is expanded based on the initial matching block and the precise motion vector to obtain the maximum matching region; Based on the maximum matching region, perform region deduplication on the current frame to be encoded to obtain the non-repeating region of the current frame to be encoded. The non-repeating regions of the current frame to be encoded are encoded, and the non-repeating regions and the vector identifiers associated with the non-repeating regions are stored in the historical data linked list.

2. The encoding method according to claim 1, characterized in that, When the interaction device is an input device in a remote desktop scenario, receiving the user's interaction information for the current frame to be encoded includes: Receive user operation information regarding the input device; When it is determined that the operation information triggers at least a portion of the pixels in the current frame to be encoded to move and / or scroll in at least one direction, the operation information is used as interaction information.

3. The encoding method according to claim 1 or 2, characterized in that, Determining the predicted change vector of the current frame to be encoded based on the interaction information includes: Predict the motion direction and distance of the current frame to be encoded based on the interaction information; The predicted change vector is determined based on the direction of motion and the distance of motion.

4. The encoding method according to claim 1, characterized in that, The step of searching a historical data list based on the pixels of the current frame to be encoded and the predicted change vector to perform motion vector search to obtain an initial matching block and precise motion vector includes: The current frame to be encoded is divided into multiple blocks according to the pixel distribution pattern, and the predicted change vector is set as the initial search vector; Select several blocks from the resulting blocks as starting points for motion vector search, and perform motion vector search within the set range of the initial search vector to obtain precise matching blocks and precise motion vectors.

5. The encoding method according to claim 4, characterized in that, When the interactive device is an input device in a remote desktop scenario, dividing the current frame to be encoded into several blocks according to the pixel distribution pattern includes: Based on the pixels of the current frame to be encoded, the current frame to be encoded is divided according to the size of the encoding unit; The content of the block is either the original pixel value or a hash value calculated based on the pixel value.

6. The encoding method according to claim 1, characterized in that, The step of expanding the matching region based on the initial matching block and the precise motion vector to obtain the maximum matching region includes: The initial matching block is expanded in the horizontal and vertical directions of the precise motion vector to determine the upper boundary, lower boundary, left boundary and right boundary, thus forming the matching boundary; On each horizontal row of the matching boundary, determine the left and right boundaries of the horizontal row; The maximum matching region is determined based on all horizontal rows whose matching region formed by the left and right boundaries is less than a preset threshold.

7. The encoding method according to claim 1, characterized in that, The step of storing the non-repeating region and the vector identifier associated with the non-repeating region into the historical data linked list includes: Obtain the region size information, pixel information, region identifier, and precise motion vector associated with the non-repeating region; The region size information, pixel information, and region identifier of the non-repeating region are stored after the latest record in the historical data linked list, and the latest record in the historical data linked list is associated with the precise motion vector associated with the non-repeating region.

8. The encoding method according to claim 1, characterized in that, Encoding the non-repeating regions of the current frame to be encoded includes: When using shallow compression technology for encoding, after removing the repetitive areas in the image, the non-repetitive areas are encoded. The repetitive areas do not need to be encoded and are marked in the bitrate. When using deep compression technology for encoding, repeating regions and vector identifiers associated with repeating regions are set in the encoder, while non-repeating regions are encoded using deep compression technology.

9. An encoding device, characterized in that, It includes at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the encoding method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the encoding method as described in any one of claims 1 to 8.

Images