A parallel coding method and system based on desktop video region of interest

Abstract

The application provides a parallel coding method and system based on a desktop video region of interest, comprising: judging whether each frame image in a desktop video sequence is a background frame; if yes, encoding a panorama of the frame to generate a panorama code stream; if not, taking a background frame closest in time before the frame as a reference frame, and dividing a region of interest according to a mouse dragging information and other methods; parallel encoding all the regions of interest to generate a region of interest code stream; packing the panorama code stream or the region of interest code stream, adding labels and time stamps, and obtaining an output code stream; judging the type of the output code stream according to the labels, if the output code stream is a panorama code stream, decoding to obtain a panorama picture; if the output code stream is a region of interest code stream, covering the decoded region of interest picture on a panorama picture of the corresponding reference frame to obtain a reconstructed picture; and taking the panorama picture or the reconstructed picture as an output picture. The method provided by the application can improve the video coding speed, reduce the coding calculation amount and transmission bandwidth.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and in particular to a parallel encoding and decoding method and system based on the region of interest in desktop video. Background Technology

[0002] Currently popular video coding standards include H.264, a new generation video coding standard jointly proposed by the International Telecommunication Union (ITU) and the International Organization for Standardization (ISO) in 2003, its upgraded version H.265, and AV1, an open-source video coding standard developed in 2018 by Google, Mozilla, Cisco, and other companies.

[0003] H.264 is a high-efficiency video coding standard widely used for high-definition and streaming video transmission. It uses video coding techniques such as motion estimation and intra-frame prediction to provide high-quality video images while reducing bitrate. H.265 is an upgrade to H.264, providing a new coding unit structure for better parallel processing and more efficient transform methods to improve coding efficiency and video quality, which is particularly important for high-resolution and ultra-high-definition video. AV1 is an open and free video coding standard that mainly includes transform coding, motion estimation, multiple reference frames, and symbol coding techniques.

[0004] Video coding technology primarily reduces temporal and spatial redundancy in image sequences, minimizing image quality degradation while reducing storage space and transmission bandwidth. Temporal redundancy refers to the redundancy between adjacent frames in a video sequence. Since the pixel content of adjacent frames is often highly similar, this similarity can be used to reduce the data size of each frame through predictive coding, thus achieving video compression. Spatial redundancy, on the other hand, refers to the redundancy between pixels within the same frame, also known as intra-frame redundancy. Because the content of pixels within the same frame is also often highly similar, this similarity can be used to reduce the data size of each frame.

[0005] Compared to natural video, motion in desktop video is more limited to a certain range, meaning a large portion of pixels remain unchanged. Therefore, encoding the entire frame using the existing methods described above would result in a significant waste of computing resources and subsequent transmission bandwidth.

[0006] Existing Solution 2 provides an encoding and decoding method and system for multiple video streams supporting Region of Interest (ROI). The technical solution involves: at the encoding end, optionally encoding the ROI, the panorama, or a combination of both, and then packaging and sending them. At the decoding end, the transmission packets are parsed, the ROI or panorama is decoded synchronously, and the ROI frames and background frames are merged in the pixel domain according to user needs. However, when merging the ROI and panorama pixels, this solution requires two frames with the same timestamp to exist simultaneously. Continuously encoding and transmitting the panorama image results in significant waste of the static portion of the desktop video, excluding the ROI.

[0007] Existing Solution 3 provides a video coding approach that combines traditional and learning methods based on regions of interest (ROIs). It obtains the background and ROI from the frame using a learning-based method, then applies different coding parameters to both, and reuses the background in video conferencing to reduce transmission bandwidth and computational resource usage. However, this solution only defines one ROI. In desktop video, more than one subject may move or refresh; multiple locations may change simultaneously, making this solution unsuitable for desktop video scenarios. Summary of the Invention

[0008] In view of this, embodiments of the present invention provide a parallel encoding and decoding method and system based on the region of interest in desktop video, so as to eliminate or improve one or more defects existing in the prior art, and solve the problem that existing video encoding technology is not suitable for desktop video and has a large waste of computing resources and transmission bandwidth.

[0009] On one hand, the present invention provides a parallel encoding and decoding method based on the region of interest in desktop video, characterized in that the method includes the following steps:

[0010] The desktop video to be processed is acquired and converted into consecutive image frames to obtain a desktop video sequence.

[0011] The system determines whether each frame in the desktop video sequence is a background frame according to the first preset method; if it is a background frame, the panorama of the background frame is encoded to generate a panoramic bitstream.

[0012] If it is not a background frame, then the background frame that precedes this frame and is closest to it in time is used as the reference frame, and the region of interest is divided according to the second preset method; the second preset method includes a division method based on mouse drag information and a division method based on traditional or machine learning; all regions of interest are encoded in parallel to generate the corresponding region of interest bitstream;

[0013] The panoramic stream or the region of interest stream is packaged and tagged with and timestamped to obtain the output stream;

[0014] The type of the output stream is determined based on the label. If it is the panoramic stream, the panoramic stream is decoded to obtain the panoramic image.

[0015] If it is the region of interest bitstream, then the region of interest bitstream is decoded and timestamp synchronized; the decoded region of interest image is overlaid on the panoramic image of the corresponding reference frame to obtain the reconstructed image;

[0016] The panoramic or reconstructed image is output to obtain the final output image.

[0017] In some embodiments of the present invention, the first preset method is one or more of the following methods: inter-frame difference, mean hashing, etc.

[0018] In some embodiments of the present invention, the method further includes:

[0019] Obtain the current frame and a preset number of frames preceding the current frame, and convert them into grayscale images;

[0020] The current frame is compared with the preset number of frames at the pixel level, and the difference value is calculated.

[0021] If the difference value is greater than a preset difference threshold, then the current frame is used as the background frame.

[0022] In some embodiments of the present invention, before dividing the region of interest according to the second preset method, the method further includes:

[0023] Compared with the reference frame, determine whether there is mouse dragging in the frame;

[0024] If the mouse dragging occurs, the mouse dragging trajectory is obtained. Based on the starting point and ending point of the mouse dragging and the position and size information of the dragged window, the motion vector and the region of interest are determined. Other regions of interest are determined using traditional or machine learning-based methods. The position and size information includes the coordinates of the upper left corner of the window, the width of the window, and the height of the window.

[0025] If the mouse dragging situation does not exist, the region of interest is determined directly using traditional or machine learning-based methods.

[0026] In some embodiments of the present invention, determining the motion vector and the region of interest based on the starting point, ending point, and position and size information of the dragged window of the mouse drag further includes:

[0027] The motion vector is obtained based on the coordinates of the starting and ending points of the mouse drag. When the window moves to the lower right, the coordinates of the four vertices of the region of interest are calculated using the following formula:

[0028] (x,y) 左上 = (x0, y0);

[0029] (x,y) 右上 = (x0 + w + x2 - x1, y0);

[0030] (x,y) 左下 = (x0, y0 + h + y2 - y1);

[0031] (x,y) 右下 = (x0+w+x2-x1, y0+h+y2-y1);

[0032] Where (x,y) 左上 (x,y) 右上 (x,y) 左下 and (x,y) 右下 The coordinates of the top left, top right, bottom left, and bottom right corners of the region of interest are respectively represented; (x0, y0) represents the coordinates of the top left corner of the window; w represents the width of the window; h represents the height of the window; (x1, y1) represents the starting coordinates of the mouse; and (x2, y2) represents the ending coordinates of the mouse.

[0033] In some embodiments of the present invention, a segmentation model is pre-trained based on a machine learning method, the desktop video sequence is input into the segmentation model, and the segmented regions of interest are output; wherein, the segmentation model adopts one of support vector machine, decision tree, and neural network.

[0034] In some embodiments of the present invention, the training method of the partitioning model includes the following:

[0035] Acquire multiple desktop video sequences, using each frame as a training sample to construct a training sample set; divide each frame into real regions of interest as labels.

[0036] Obtain the initial model, input the training sample set one by one or in batches into the initial model, and output the predicted region of interest for the corresponding sample;

[0037] The initial model is trained using the training sample set to construct the loss between the predicted region of interest and the label, and the model is optimized until the preset performance requirements are met to obtain a partitioning model for dividing the region of interest.

[0038] On the other hand, the present invention provides a parallel encoding and decoding system based on the region of interest in desktop video, characterized by comprising the following:

[0039] The video processing module is used to acquire the desktop video to be processed and convert the desktop video into continuous image frames to obtain a desktop video sequence;

[0040] The encoder and decoder are used to perform the steps of the parallel encoding and decoding method based on the region of interest of desktop video as described in any of the above descriptions, in order to encode and decode the desktop video.

[0041] The storage module is used to store the desktop video, desktop video sequence, encoder and decoder outputs;

[0042] The controller is used to control the video processing module, the encoder, the decoder and the storage module to perform corresponding functions.

[0043] On the other hand, the present invention provides an electronic device including a processor and a memory, characterized in that the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory, wherein when the computer instructions are executed by the processor, the electronic device implements the steps of the parallel encoding and decoding method based on the region of interest of desktop video as described above.

[0044] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the steps of the parallel encoding and decoding method based on the region of interest of desktop video as described in any of the above.

[0045] The beneficial effects of the present invention are at least as follows:

[0046] This invention provides a parallel encoding and decoding method and system based on regions of interest (ROIs) in desktop video, comprising: acquiring a desktop video sequence; determining whether each frame in the desktop video sequence is a background frame; if so, performing panoramic encoding on that frame to generate a panoramic bitstream; if not, using the nearest background frame preceding that frame as a reference frame, determining whether mouse dragging information exists, and using mouse dragging information and / or traditional or machine learning-based methods to divide the ROI; processing each ROI separately to achieve parallel encoding and generate an ROI bitstream; packaging the panoramic bitstream or ROI bitstream and adding tags and timestamps to obtain an output bitstream; determining the type of the output bitstream based on the tags; if it is a panoramic bitstream, decoding to obtain a panoramic image; if it is an ROI bitstream, overlaying the decoded ROI image onto the panoramic image of the corresponding reference frame to obtain a reconstructed image; and using the panoramic image or reconstructed image as the output image. The method provided by this invention reduces the encoding frequency of panoramic images, encodes ROIs with lower resolution and lower encoding overhead, improves video encoding speed while reducing the performance overhead required for encoding, reducing the amount of encoding computation and the required transmission bandwidth, and greatly reducing waste.

[0047] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0048] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0050] Figure 1 This is a schematic diagram illustrating the steps of a parallel encoding and decoding method based on the region of interest in desktop video in one embodiment of the present invention.

[0051] Figure 2 This is a flowchart of a parallel encoding method based on the region of interest in desktop video in one embodiment of the present invention.

[0052] Figure 3 This is a flowchart of a decoding method based on the region of interest in desktop video according to an embodiment of the present invention.

[0053] Figure 4 This refers to two temporally consecutive frames in a desktop video sequence in one embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0055] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0056] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0057] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0058] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0059] It should be emphasized here that the step markers mentioned below are not a limitation on the order of the steps, but should be understood as meaning that the steps can be executed in the order mentioned in the embodiments, or in a different order than in the embodiments, or several steps can be executed simultaneously.

[0060] To address the limitations of existing video encoding technologies for desktop video, which result in significant waste of computing resources and transmission bandwidth, this invention provides a parallel encoding and decoding method based on the region of interest (ROI) in desktop video, such as... Figure 1 As shown, the method includes the following steps S101 to S105:

[0061] Step S101: Obtain the desktop video sequence to be processed.

[0062] Step S102: Determine whether each frame in the desktop video sequence is a background frame according to the first preset method; if it is a background frame, encode the panorama of the background frame to generate a panoramic bitstream.

[0063] Step S103: If it is not a background frame, then the background frame that is closest to this frame in time before this frame is used as the reference frame, and the region of interest is divided according to the second preset method; wherein, the second preset method includes a division method based on mouse drag information and a division method based on traditional or machine learning; all regions of interest are encoded in parallel to generate the corresponding region of interest bitstream.

[0064] Step S104: Package the panoramic bitstream or region of interest bitstream, and add tags and timestamps to obtain the output bitstream.

[0065] Step S105: Determine the type of the output bitstream based on the tag. If it is a panoramic bitstream, decode the panoramic bitstream to obtain the panoramic image.

[0066] Step S106: If it is a region of interest bitstream, decode the region of interest bitstream and synchronize the timestamp; overlay the decoded region of interest image onto the panoramic image of the corresponding reference frame to obtain the reconstructed image.

[0067] Step S107: Output the panoramic or reconstructed image to obtain the final output image.

[0068] like Figure 2The flowchart shown is a parallel encoding method based on the region of interest in desktop video, which includes steps S101 to S104.

[0069] In step S101, the desktop video sequence to be processed is first obtained, wherein the desktop video sequence is in YUV format and can be directly read by the encoder.

[0070] In some embodiments, the desktop video is preprocessed before conversion to obtain higher quality image frames. Exemplary preprocessing includes operations such as filtering.

[0071] In step S102, each frame in the desktop video sequence is determined to be a background frame. In this invention, the determination of a background frame is based on the magnitude of the change in image content between the current frame and the previous frame.

[0072] In some embodiments, it is determined whether each frame in the desktop video sequence is a background frame according to a first preset method. The first preset method is one or more of the following methods: inter-frame differencing, mean hashing, etc. Specifically:

[0073] Obtain the current image frame to be judged and a preset number of previous frames, and convert them into grayscale images.

[0074] The grayscale image of the current frame is compared with the grayscale images of a preset number of frames to calculate the difference value. The difference value is compared with a preset difference threshold. If the difference value is greater than the difference threshold, the current frame is used as the background frame. If the difference value is less than the difference threshold, the process proceeds directly to step S103.

[0075] Alternatively, a hash operation can be performed on the grayscale image of the current frame and the grayscale images of a preset number of frames to calculate the hash value. The Hamming distance between the hash values ​​of the current frame and the preset number of frames can be calculated to obtain the amount of change in the image content between the two frames. The amount of change in the image content is compared with a preset change threshold. If the amount of change in the image content is greater than the change threshold, the current frame is used as the background frame. If the amount of change in the image content is less than the change threshold, the process proceeds directly to step S103.

[0076] In step S103, based on the judgment in step S102, if the current frame is not a background frame, then a region of interest (ROI) is defined from the current frame. The ROI refers to the part of an image or video that the user or algorithm is interested in. In this invention, considering that motion in desktop videos is mostly within a certain range and often involves changes in relatively fixed structures such as windows, the changing areas are designated as ROIs. In non-background frames, only the ROIs are encoded and transmitted to improve encoding speed and reduce computational load and required transmission bandwidth.

[0077] In some embodiments, before dividing the region of interest (ROI), it is first determined whether there is mouse dragging in the current frame. Since desktop videos often contain relatively fixed structures such as windows, mouse dragging plays a crucial role in changes to the screen content. Based on mouse dragging information, the ROI can be divided more easily. Specifically:

[0078] First, determine if there is mouse dragging in the current frame. If there is mouse dragging, obtain the mouse dragging trajectory. Based on the starting point and ending point of the mouse dragging and the position and size information of the dragged window, determine the motion vector and region of interest. Then, use traditional or machine learning-based methods to determine other regions of interest. The position and size information includes at least the coordinates of the top left corner of the window, the width of the window, and the height of the window.

[0079] If there is no mouse dragging, the region of interest is determined directly using traditional or machine learning-based methods.

[0080] In some embodiments, conventional methods refer to conventional image processing methods, such as edge detection algorithms, region segmentation algorithms, region selection algorithms, feature extraction algorithms, etc.

[0081] In some embodiments, the machine learning-based method involves inputting a desktop video sequence into a pre-trained partitioning model and outputting partitioned regions of interest. For example, the partitioning model may employ one of support vector machines, decision trees, or neural networks.

[0082] In some embodiments, the training method for the partitioning model includes the following:

[0083] Acquire multiple desktop video sequences, using each frame as a training sample to construct a training sample set; and divide each frame into real regions of interest as labels.

[0084] Obtain the initial model, input the training sample set one by one or in batches into the initial model, and output the region of interest of the corresponding sample obtained from the prediction.

[0085] The initial model is trained using a training sample set to construct the loss between the predicted region of interest and the label. The model is then optimized until the preset performance requirements are met, resulting in a partitioning model for dividing the region of interest.

[0086] In some embodiments, if mouse dragging occurs, the dragging trajectory of the mouse is obtained, and the motion vector and region of interest are determined based on the starting point, ending point, and position and size information of the dragged window. Specifically, assuming the starting point coordinates of the mouse dragging to the lower right are (x1, y1) and the ending point coordinates are (x2, y2), the position and size information of the window includes the following: the coordinates of the upper left corner of the window are (x0, y0), the width of the window is w, and the height of the window is h. The motion vector of the window is determined based on the starting and ending coordinates of the mouse, and the coordinates of the four vertices of the region of interest are calculated based on the motion vector when the window moves to the lower right, as shown in formulas (1) to (4).

[0087] (x,y) 左上 = (x0, y0); (1)

[0088] (x,y) 右上 =(x0+w+x2-x1,y0); (2)

[0089] (x,y) 左下 =(x0,y0+h+y2-y1); (3)

[0090] (x,y) 右下 =(x0+w+x2-x1,y0+h+y2-y1); (4)

[0091] Among them, (x,y( 左上 (x,y) 右上 (x,y) 左下 and (x,y) 右下 These represent the coordinates of the upper left, upper right, lower left, and lower right corners of the region of interest, respectively.

[0092] Each region of interest is encoded individually, that is, all regions of interest are encoded in parallel to generate the corresponding region of interest bitstream.

[0093] In step S104, the panoramic stream or the stream of interest is packaged and tagged with a timestamp to obtain the output stream. For example, if it is a panoramic stream, the tag is 1; if it is a stream of interest, the tag is 0.

[0094] like Figure 3 The diagram shows a flowchart of a decoding method based on the region of interest in desktop video, which includes steps S105 to S107.

[0095] In step S105, the output bitstream obtained in step S104 is acquired, and it is determined whether the output bitstream is a panoramic bitstream or a bitstream of interest based on the tag. If it is a panoramic bitstream, it is directly decoded using a decoder to obtain the panoramic image.

[0096] In step S106, if the stream of interest is determined to be a stream of interest, the stream of interest is first decoded to determine the reference frame of the image frame corresponding to the stream of interest, and the panoramic image of the reference frame after decoding is obtained. Based on the timestamp, the image of the region of interest obtained at the same time as the above decoding is overlaid on the corresponding position of the panoramic image of the reference frame to obtain the reconstructed image.

[0097] In step S107, the obtained panoramic or reconstructed image is output to obtain the final output image, thus realizing the encoding and decoding operation based on the region of interest of desktop video.

[0098] The parallel encoding and decoding method based on the region of interest (ROI) of desktop video provided by the present invention will be further described below with reference to a specific embodiment:

[0099] In this embodiment, as Figure 4 As shown, taking two temporally consecutive frames as an example, Figure 4 The image frame on the left is designated as the first frame, the image frame on the right is designated as the second frame, and the first frame is designated as the background frame.

[0100] Compare Figure 4 As seen in the two frames, the browser window moved a certain distance to the upper right in the desktop video. The pre-defined encoder detected a mouse dragging action. Therefore, using the mouse dragging information and the window's position and size information, a region of interest (represented by the box on the left in the second frame) was defined. Then, other regions of interest (represented by the box on the right in the second frame) were divided using traditional or machine learning-based methods. Thus, two regions of interest were obtained in the second frame.

[0101] Since the first frame is a background frame, it is directly encoded panoramically to generate a panoramic bitstream. The panoramic bitstream is then decoded to obtain the panoramic image. In parallel, the two regions of interest (ROIs) in the second frame are encoded, generating two corresponding ROIs. These two ROIs are then packaged and timestamped. The ROIs are then decoded to obtain the corresponding ROI images. Based on the timestamped images, these two ROI images are overlaid onto the corresponding positions of the panoramic image in the first frame (i.e., ...). Figure 4 The corresponding box range in the second frame is the coverage area, and the reconstructed image is obtained.

[0102] Experimental tests were conducted on the parallel encoding and decoding method based on the Region of Interest (ROI) of desktop video provided by this invention. Specifically, the impact of dividing the ROI into different sizes (resolutions) on the encoding time of one frame was tested. The tested video sequence had a resolution of 1920*1080 and contained a total of 1500 frames. Assuming that the number of background frames in the video was 3 (corresponding to Base in Table 1), then only these 3 frames needed to be encoded at 1080P resolution. The experimental data are shown in Table 1.

[0103] Table 1

[0104]

[0105]

[0106] As shown in Table 1, when the region of interest is relatively small, the method provided by this invention can effectively reduce the encoding time.

[0107] The present invention also provides a parallel encoding and decoding system based on the region of interest (ROI) of desktop video, comprising the following:

[0108] The video processing module is used to acquire the desktop video to be processed and convert the desktop video into continuous image frames to obtain a desktop video sequence.

[0109] The encoder and decoder are used to perform steps of a parallel encoding and decoding method based on the region of interest of the desktop video to encode and decode the desktop video.

[0110] The storage module is used to store the desktop video, the desktop video sequence, and the output results of the encoder and the decoder;

[0111] The controller is used to control the video processing module, encoder, decoder, and storage module to perform their respective functions.

[0112] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the electronic device implements the steps of a parallel encoding and decoding method based on a region of interest in desktop video.

[0113] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a parallel encoding and decoding method based on a region of interest in desktop video.

[0114] Corresponding to the above method, the present invention also provides an apparatus comprising a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus performs the steps of the method as described above.

[0115] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0116] In summary, this invention provides a parallel encoding and decoding method and system based on regions of interest (ROIs) in desktop video, comprising: acquiring a desktop video sequence; determining whether each frame in the desktop video sequence is a background frame; if so, performing panoramic encoding on that frame to generate a panoramic bitstream; if not, using the preceding and nearest background frame as a reference frame, determining whether mouse dragging information exists, and using mouse dragging information and / or traditional or machine learning-based methods to divide the ROI; processing each ROI separately to achieve parallel encoding and generate an ROI bitstream; packaging the panoramic bitstream or ROI bitstream and adding tags and timestamps to obtain an output bitstream; determining the type of the output bitstream based on the tags; if it is a panoramic bitstream, decoding to obtain a panoramic image; if it is an ROI bitstream, overlaying the decoded ROI image onto the panoramic image of the corresponding reference frame to obtain a reconstructed image; and using the panoramic image or reconstructed image as the output image. The method provided by this invention reduces the encoding frequency of panoramic images, encodes ROIs with lower resolution and lower encoding overhead, improves video encoding speed while reducing the performance overhead required for encoding, reducing the amount of encoding computation and the required transmission bandwidth, and greatly reducing waste.

[0117] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0118] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0119] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A parallel encoding and decoding method based on regions of interest in desktop video, characterized in that, The method includes the following steps: Obtain the desktop video sequence to be processed; The system determines whether each frame in the desktop video sequence is a background frame according to the first preset method; if it is a background frame, the panorama of the background frame is encoded to generate a panoramic bitstream. If it is not a background frame, the background frame that is closest to the current frame in time before the current frame is used as a reference frame. The current frame is compared with the reference frame to determine whether there is mouse dragging in the current frame. If there is mouse dragging, the mouse dragging trajectory is obtained. Based on the start and end points of the mouse dragging and the position and size information of the dragged window, the motion vector and region of interest are determined. Then, other regions of interest are determined using traditional or machine learning-based methods. If there is no mouse dragging, the regions of interest are determined directly using traditional or machine learning-based methods. All regions of interest are encoded in parallel to generate the corresponding region of interest bitstream. The position and size information includes the coordinates of the upper left corner of the window, the width and height of the window. The panoramic stream or the region of interest stream is packaged and tagged with and timestamped to obtain the output stream; The type of the output stream is determined based on the label. If it is the panoramic stream, the panoramic stream is decoded to obtain the panoramic image. If it is the region of interest bitstream, then the region of interest bitstream is decoded and timestamp synchronized; the decoded region of interest image is overlaid on the panoramic image of the corresponding reference frame to obtain the reconstructed image; The panoramic or reconstructed image is output to obtain the final output image.

2. The parallel encoding and decoding method based on the region of interest in desktop video according to claim 1, characterized in that, The first preset method is one or more of the following methods: inter-frame difference, mean hashing, etc.

3. The parallel encoding and decoding method based on the region of interest in desktop video according to claim 2, characterized in that, Also includes: Obtain the current frame and a preset number of frames preceding the current frame, and convert them into grayscale images; The current frame is compared with the preset number of frames at the pixel level, and the difference value is calculated. If the difference value is greater than a preset difference threshold, then the current frame is used as the background frame.

4. The parallel encoding and decoding method based on the region of interest in desktop video according to claim 1, characterized in that, Based on the start and end points of the mouse drag and the position and size information of the dragged window, the motion vector and the region of interest are determined, and the method further includes: Based on the coordinates of the start and end points of the mouse drag, the motion vector is obtained. Based on this motion vector, when the window moves to the lower right, the coordinates of the four vertices of the region of interest are calculated as follows: ； ； ； ； in, , , and These represent the coordinates of the upper left, upper right, lower left, and lower right corners of the region of interest, respectively. This represents the coordinates of the top-left corner of the window; This indicates the width of the window; Indicates the height of the window; Indicates the starting coordinates of the mouse cursor; This indicates the coordinates of the mouse's endpoint.

5. The parallel encoding and decoding method based on the region of interest in desktop video according to claim 1, characterized in that, A segmentation model is pre-trained based on a machine learning method. The desktop video sequence is input into the segmentation model, and the segmented regions of interest are output. The segmentation model adopts one of support vector machines, decision trees, and neural networks.

6. The parallel encoding and decoding method based on the region of interest in desktop video according to claim 5, characterized in that, The training method for the partitioning model includes the following: Acquire multiple desktop video sequences, and use each frame as a training sample to construct a training sample set; For each frame of the image, a real region of interest is defined, and the defined real region of interest is used as a label. Obtain the initial model, input the training sample set one by one or in batches into the initial model, and output the predicted region of interest for the corresponding sample; The initial model is trained using the training sample set to construct the loss between the predicted region of interest and the label, and the model is optimized until the preset performance requirements are met to obtain a partitioning model for dividing the region of interest.

7. A parallel encoding and decoding system based on regions of interest in desktop video, characterized in that, Includes the following: The video processing module is used to acquire the desktop video to be processed and convert the desktop video into continuous image frames to obtain a desktop video sequence; An encoder and a decoder are used to perform the steps of the method as described in any one of claims 1 to 6 to encode and decode desktop video. The storage module is used to store the desktop video, the desktop video sequence, and the output results of the encoder and the decoder; The controller is used to control the video processing module, the encoder, the decoder and the storage module to perform corresponding functions.

8. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the electronic device implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images