Video processing method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a video processing method and device, electronic equipment and a readable storage medium, and the method applied to a server comprises the steps: carrying out the blocking of each video frame, and obtaining a block image; wherein the block images at the same position in each video frame respectively have a corresponding relationship; determining a target block image according to the similarity between the current block image and the forward block image; wherein the current block image is a block image in the current video frame, and the forward block image is a block image corresponding to the current block image in the previous video frame; and independently coding the target block image according to a preset coding format to obtain a coding result of the target block image, and sending the coding result to the client, so that when the current block image of which the similarity is greater than a preset threshold exists in the target video generated by the client, a forward block image corresponding to the current block image is displayed. The video processing amount can be reduced, so that the video processing speed is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a video processing method, a video processing apparatus, an electronic device, and a readable storage medium. Background Technology

[0002] Currently, servers running virtual machines typically use a Central Processing Unit (CPU) for graphics processing and video encoding / decoding to meet the data processing needs of multiple virtual machines during video playback. Understandably, video playback involves a large number of encoding / decoding computation threads, placing high demands on the server's computing power.

[0003] To ensure video transmission speed, hardware devices capable of supporting high computational demands are currently required. For example, physical graphics cards need to be installed in the aforementioned servers for video processing; otherwise, excessive CPU load will significantly reduce video processing speed. Summary of the Invention

[0004] In view of the above problems, embodiments of this application are proposed to provide a video processing method that overcomes or at least partially solves the above problems, which can reduce the amount of computation in the video data transmission process, that is, reduce the hardware requirements for video transmission when remotely accessing virtual machines, thereby improving the speed of video processing.

[0005] Accordingly, embodiments of this application also provide a video processing apparatus, an electronic device, and a readable storage medium to ensure the implementation and application of the above methods.

[0006] In a first aspect, embodiments of this application disclose a video processing method applied to a server, the method comprising: Each video frame is divided into blocks to obtain block images; wherein, there is a correspondence between block images with the same position in each video frame; The target block image is determined based on the similarity between the current block image and the forward block image; wherein, the current block image is the block image in the current video frame, and the forward block image is the block image in the previous video frame that corresponds to the current block image; The target segment image is independently encoded according to a preset encoding format to obtain the encoding result of the target segment image and send it to the client. When the target video generated by the client contains a current segment image with a similarity greater than a preset threshold, the forward segment image corresponding to the current segment image is displayed.

[0007] Secondly, embodiments of this application provide a video processing method applied to a client, the method comprising: The server receives the encoding result of the target block image sent by the server; the server divides each video frame into blocks to obtain block images; the target block image is determined by the server based on the similarity between the current block image and the previous block image; the current block image is the block image in the current video frame, and the previous block image is the block image in the previous video frame that corresponds to the current block image; the encoding result is obtained by the server independently encoding the target block image according to a preset encoding format. The encoding result of the target block image is decoded according to the preset encoding format to obtain the target block image; The target video is obtained based on the target segmented image; wherein, when there is a current segmented image with a similarity greater than a preset threshold, the forward segmented image corresponding to the current segmented image is displayed.

[0008] Thirdly, embodiments of this application disclose a video processing apparatus applied to a server, the apparatus comprising: The segmentation module is used to segment each video frame into blocks and obtain segmented images; wherein, there is a corresponding relationship between segmented images with the same position in each video frame. The selection module is used to determine the target block image based on the similarity between the current block image and the previous block image; wherein, the current block image is the block image in the current video frame, and the previous block image is the block image in the previous video frame that corresponds to the current block image; The encoding module is used to independently encode the target segment image according to a preset encoding format, obtain the encoding result of the target segment image and send it to the client, so that when the target video generated by the client contains a current segment image with a similarity greater than a preset threshold, the forward segment image corresponding to the current segment image is displayed.

[0009] Fourthly, embodiments of this application disclose a video processing apparatus applied to a client, the apparatus comprising: A receiving module is used to receive the encoding result of the target block image sent by the server; the server divides each video frame into blocks to obtain block images; the target block image is determined by the server based on the similarity between the current block image and the preceding block image; the current block image is the block image in the current video frame, and the preceding block image is the block image in the previous video frame corresponding to the current block image; the encoding result is obtained by the server independently encoding the target block image according to a preset encoding format; A decoding module is used to decode the encoding result of the target block image according to the preset encoding format to obtain the target block image; The generation module is used to obtain a target video based on the target segmented image; wherein, when there is a current segmented image with a similarity greater than a preset threshold, the forward segmented image corresponding to the current segmented image is displayed.

[0010] Fifthly, embodiments of this application disclose an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform the steps of any of the video processing methods described above.

[0011] Sixthly, embodiments of this application disclose a readable storage medium storing a program or instructions that, when executed by a processor, can implement any of the video processing methods described above.

[0012] This application provides a video processing method in which the server divides video frames into blocks to obtain block images; the block images are then compared with the previous video frame for similarity, and the block images with lower similarity are determined as target block images. Only the target block images are encoded and sent to the client. In other words, by reusing the encoding results of block images with higher similarity, the number of video frames or block images within video frames that need to be encoded is reduced, thereby reducing the demand for encoding and decoding calculations during video transmission and improving the speed of video encoding and decoding.

[0013] Specifically, in application, the server only encodes and transmits image blocks with low similarity (e.g., similarity less than a preset threshold), ignoring highly similar blocks. Image blocks requiring encoding are encoded independently according to a preset encoding format. After decoding the acquired encoding results, the client can play the video frames in a way that allows the next frame to overwrite the previous frame. Therefore, when the current image block data is missing, the previous frame's corresponding forward image block will still be displayed in the missing space, effectively reusing the image block at the same position from the previous frame. This reduces the computational load for each video frame during encoding and decoding, thereby increasing video processing speed. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a video processing architecture according to this application; Figure 2 This is a flowchart illustrating the steps of an embodiment of a video processing method according to this application; Figure 3 This is a schematic diagram of an embodiment of a video sorting method according to this application; Figure 4This is a flowchart illustrating the steps of another embodiment of the video processing method of this application; Figure 5 This is a schematic flowchart of a video processing method according to an embodiment of this application; Figure 6 This is a structural block diagram of one embodiment of video processing in this application; Figure 7 This is a structural block diagram of another video processing embodiment of this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] In applications that remotely access virtual machines (such as cloud desktop applications), multiple virtual machines are needed to play videos. Servers hosting virtual machines typically do not have dedicated graphics cards or high-performance integrated graphics cards, relying solely on the CPU for graphics processing and video encoding / decoding. The large amount of encoding / decoding computation places high demands on the server's computing power. In cloud desktop scenarios where no graphics card is configured and only the CPU is used for encoding / decoding computation, the CPU can easily become fully occupied when the number of virtual machines increases, causing video playback stuttering.

[0017] The core technology of cloud desktop applications relies on virtual machines and remote video streaming, and video playback is one of the key aspects of the user experience of cloud desktop applications.

[0018] Reference Figure 1 The diagram illustrates a video transmission process that can be applied to cloud desktop applications.

[0019] Cloud desktop applications involve two types of physical devices: servers and clients. Virtual machines and servers are deployed on the server, and users can remotely access the virtual machines deployed on the server through clients (cloud terminal devices). When a virtual machine plays a video, the server obtains video image frames (i.e., raw image data) from the virtual graphics card of the virtual machine. These video image frames are encoded and then encapsulated into a suitable encapsulation format and streamed to the network. They are then transmitted to various clients through the network. The clients decode the video image frames and display them locally, thus forming video playback.

[0020] In this embodiment, the raw image data is in Blue, Green, Red, X (Unused / Reserved), BGRX format, which is a common 24 / 32-bit pixel format mainly used for video processing, camera acquisition, and computer vision.

[0021] Reference Figure 2 The diagram illustrates a flowchart of an embodiment of a video processing method according to this application. The method is applied to a server and may include the following steps: Step 101: Divide each video frame into blocks and obtain block images; wherein, there is a corresponding relationship between block images with the same position in each video frame; Step 102: Determine the target block image based on the similarity between the current block image and the forward block image; wherein, the current block image is the block image in the current video frame, and the forward block image is the block image in the previous video frame that corresponds to the current block image; Step 103: Encode the target block image independently according to the preset encoding format, obtain the encoding result of the target block image and send it to the client, so that when the target video generated by the client contains a current block image with a similarity greater than the preset threshold, the forward block image corresponding to the current block image is displayed.

[0022] The video processing method provided in this application can be applied to the server to reduce the computational load of video encoding and decoding, thereby improving data processing efficiency.

[0023] In this embodiment, each video frame is segmented to obtain segmented images, and there is a correspondence between segmented images at the same position. The segmented image in the current video frame is the current segmented image, and the segmented image in the previous video frame is the forward segmented image. The similarity between the current segmented image and the forward segmented image is obtained, and video frame data with low similarity (i.e., target segmented images) is selected for encoding and transmission, reducing the amount of data that needs to be encoded and transmitted (highly similar video frame data), thereby reducing the corresponding hardware requirements.

[0024] Understandably, the target segmented image is obtained by comparing the similarity of the current segmented image in the current video frame with the forward segmented image in the previous video frame. For the first frame of the video, since there is no previous video frame and no reference point for comparison, the entire video frame is used as the target segmented image. That is, the first frame of all videos is the target segmented image, and subsequent target segmented images are confirmed by comparing their similarity with the segmented image at the corresponding position in the previous frame. Since consecutive video frames usually have high similarity, it is unnecessary to encode and send segments with high similarity, thus reducing the amount of data that needs to be transmitted in this scheme.

[0025] Specifically, the video frame is divided into multiple image blocks, and the similarity between different image blocks is independently compared with the image blocks in the same position in the previous video frame, with the comparison results not affecting each other. Compared with comparing the complete current video frame with the complete previous video frame, this method can obtain more conclusions about high similarity between images.

[0026] For example, such as Figure 3 The image shows two adjacent video frames. When regions a3 and b3 are identical (i.e., similarity higher than a preset threshold), and a1, a2, and a4 are different from b1, b2, and b4 respectively (i.e., similarity lower than a preset threshold), the complete a-frame and b-frame are generally considered different. In other words, the similarity between the complete a-frame and b-frame is usually less than the preset threshold. At this point, the current block image with a similarity greater than or equal to the preset threshold is 0. If b-frame is taken as the current frame, then the complete b-frame is the target block image to be encoded and transmitted. However, if the complete video frame is divided into blocks, resulting in block images, and these blocks are compared in groups, for example, four groups of block images are compared independently, where the similarity between a3 and b3 is greater than the preset threshold, meaning there exists a current block image (b3) with a similarity greater than the preset threshold to the corresponding preceding block image (a3). In this case, for frame b, the target block image only includes b1, b2, and b4. Therefore, when dividing a video frame into blocks and comparing the similarity of blocks with the same (i.e., corresponding) positions, more current block images with similarity greater than a preset threshold will appear. That is, when dividing a complete video frame into blocks and comparing the blocks, block images with high similarity to the corresponding positions of the previous frame may appear. Therefore, the number of target block images (i.e., current block images with low similarity) that need to be encoded independently will be reduced.

[0027] Accordingly, in the above-described situation, according to the implementation method provided in this application, when calculating the similarity based on complete a-frames and b-frames, the forward block image is frame a, and the current block image is frame b. Since there is no current block image whose similarity meets the preset threshold, it is necessary to encode and send the complete a-frame and complete b-frame to the client separately. However, when dividing the complete video frame image into blocks and comparing the similarity of blocks at the same position, since the similarity between the forward block image a3 and the current block image b3 is greater than the preset threshold, the image data of b3 will not be processed; only the image data of a1-a4, b1, b2, and b4 will be encoded and transmitted. This reduces the amount of image data that needs to be processed and improves the video processing speed.

[0028] The number of image blocks in a single video frame can be an integer greater than 0, or it can be 1, meaning that the video frame can be treated as a single image block without segmentation. For example, if the video content is a still life shot (such as a time-lapse video of a natural scene shot from a fixed point), the original video frames have a high degree of similarity. There is no need to segment the video frames to increase the probability of high-similarity image blocks between them; segmentation would only introduce redundant computation.

[0029] To address this, the number of image blocks can be set by the user as needed. For example, after confirming the video to be transmitted, the user or relevant technicians can observe the video content and select / input an appropriate number of image blocks based on experience. Alternatively, the user or relevant technicians can determine the number of image blocks based on the type of video to be transmitted. For instance, compared to a fixed-angle surveillance video, which typically has less variation between adjacent frames, and a competitive game video, which typically has greater variation between adjacent frames, the former already has a high degree of similarity between adjacent frames. Therefore, it is not necessary to divide the video frame images into blocks to increase the probability of highly similar image blocks appearing. Thus, the former usually requires fewer image blocks than the latter. In other words, different video types correspond to different numbers of image blocks.

[0030] Understandably, the number of image blocks can be adjusted by the segmentation program based on the similarity calculation results between pixels in multiple randomly selected video frames of the current video. For example, multiple adjacent pixels with high similarity between different video frames can be grouped into one image block, while multiple adjacent pixels with low similarity between different video frames can be grouped into another image block, thus determining the number of image blocks based on the similarity between sampled video frames. Alternatively, the segmentation program can determine the number of image blocks according to preset segmentation principles, such as uniformly setting the number of image blocks required for all video frames; or performing segmentation according to the video source (surveillance camera or video database, etc.) / video type, etc., based on the corresponding preset number of image blocks (for example, if the video is from a surveillance camera, the similarity between adjacent video frames is high by default, so the number of image blocks required is usually less than that of videos from a video database).

[0031] Optionally, the step of dividing each video frame into blocks and obtaining block images includes: Each video frame is divided into blocks according to a preset n×m specification, and at least one block image of the n×m specification corresponding to each video frame is obtained; where n and m are positive integers.

[0032] In other words, video frames can be divided into blocks using a grid partitioning method to obtain block images of a preset size. The block images obtained by the grid partitioning method have stable shapes, making it easy to compare the block images in the current video frame with those in the previous video frame. At the same time, the grid partitioning method has simple logic, requires little computation, and does not impose computational pressure, which can further reduce the computational requirements of the video encoding and decoding process.

[0033] As is understandable, image gridding is a method of dividing an image into multiple smaller blocks, commonly used in image processing and analysis. This method, by dividing the image into multiple grid blocks, allows for convenient independent processing or analysis of each grid block. First, the original image to be processed needs to be read. Second, the image is divided into multiple n×m grid blocks according to the requirements. The coordinates of each grid block within the entire image can be determined based on the size of the grid block (n and m) (e.g., the coordinates of the four corners of each grid block). Then, based on the coordinates of the aforementioned grid blocks, the original image is cropped to obtain each grid block. Finally, each cropped grid block is stored or processed separately.

[0034] It is understandable that the values ​​of n and m can be different or the same, meaning that the final segmented image can be either rectangular or square.

[0035] It should be noted that if the video frame cannot be divided into multiple complete image blocks due to the resolution not matching the preset n×m specification, the edge processing mode will be enabled. For example, if any one of the four corners of the original video frame image is set as the origin of the plane coordinate system, and the original video frame image is divided into blocks according to the n×m specification starting from the origin, there will be multiple remaining regions that do not meet the n×m specification in positions far from the origin. In this case, the edge processing mode will be applied to these multiple remaining regions.

[0036] The edge processing modes include: Mode 1: Retaining the multiple remaining regions and treating them as special block images. In this case, the special block image in the current video frame still corresponds completely to the position of the special block image in the previous video frame. The remaining regions are then used as special block images in subsequent processes such as similarity calculation, target block image determination, and encoding and transmission. Mode 2: Merging the multiple remaining regions with the adjacent block images. The merged image is then used as a special block image in subsequent processes such as similarity calculation, target block image determination, and encoding and transmission. It should be clarified that in Mode 2, the logic of merging the remaining regions with adjacent block images in all video frames is consistent, that is, ensuring that the special block image in the current video frame still corresponds completely to the position and size of the special block image in the previous video frame.

[0037] Specifically, the n×m specification can be chosen to be 128×128 or 64×64, primarily for 1080P (1920×1080) video frames. Understandably, the smaller the n×m specification (i.e., the smaller the values ​​of n and m), the more similar image blocks there are between adjacent video frames. This means a larger area of ​​images in the current video frame that are determined not to require encoding and transmission (e.g., the comparison between the complete b-frame and the segmented b1-b4 images in the previous example). The proportion of the area of ​​the further target image blocks relative to the complete video frame will decrease, meaning less data needs to be encoded and transmitted.

[0038] For example, when n and m are the length and width of the current video frame, respectively, meaning the segmented image is a complete video frame image, the complete current video frame image has low similarity to the complete previous video frame image; therefore, the complete current video frame image is always the target segmented image. However, when the values ​​of n and m are halved, causing the size of the segmented image to change from the size of the complete video frame n×m to n'×m', where n' is... , m' is At that time, the complete current video frame image (such as...) Figure 3 The b-frame shown will be divided into four equal-sized blocks (n'×m'), named b1, b2, b3, and b4; at this point, the complete previous video frame image (such as...) will be... Figure 3 The frame shown (a) will also be divided into 4 blocks in the same way, namely a1, a2, a3, and a4. If there is a case where a3 and b3 have high similarity and the remaining parts have low similarity, then the target block image in frame b will be b1, b2, and b4. That is, the b3 part does not need to be processed (encoded and sent). Compared with the case of using the complete video frame as block images, the amount of image data that needs to be processed is reduced.

[0039] It should be noted that the smaller the values ​​of n and m are, the greater the pressure on image segmentation, meaning that more image blocks need to be divided, and jagged (blocky) artifacts are more likely to be generated at the boundaries between blocks in the image blocks, affecting the viewing experience. Therefore, the number of image blocks needs to be adjusted according to actual needs to avoid extreme values.

[0040] In some cases, the values ​​of n and m can be adjusted based on the size of the complete video frame to ensure that the n×m format can completely divide the video frame. This means that the length and width of the complete image frame are integer multiples of n and m, with no edges that cannot be completely divided. For example, if the original video frame is 720×480, n can be set to 360 and m to 240, so that the current video frame can be completely divided into four identical 360×240 image blocks. However, if the original video frame is 1920×1080, meaning 360×240 is not divisible by it, then n and m will be adjusted to 384×216, based on the size of the complete image frame, so that the 1920×1080 original video frame can be completely divided into 25 identical image blocks.

[0041] Alternatively, images can be segmented in other ways. For example, an adaptive segmentation method can be applied to the first video frame, dividing the video frame into blocks according to its image content. Then, all remaining video frames are segmented into blocks with the same regions as the first video frame, ensuring that all video frames correspond perfectly to the shape, size, and position of the segmented regions in the previous frame, resulting in multiple segmented images. Specifically, the adaptive segmentation method uses local features (RGB color parameters, grayscale values, brightness values, etc.) to obtain finer segments for complex regions, while simple regions are processed into larger, uniform blocks. Examples of adaptive segmentation methods that can be referenced include region growing and quadtree decomposition. It is understood that the above methods are merely illustrative and do not limit the types of adaptive segmentation methods that can be used.

[0042] Among them, the region growing method first grids the image according to preset rules (e.g., dividing it every p pixels), then calculates the feature differences (such as color differences) between the current region and its neighbors after division. If the difference is less than a threshold T, the regions are merged. This process iteratively merges regions that meet the conditions until no new regions that meet the conditions can be merged, thus obtaining the final division result. The quadtree decomposition method first uses the entire video frame as the initial block and calculates the uniformity measure (such as pixel value variance) of the current block (initial block). If the uniformity measure indicates that the current image is not uniform (e.g., variance σ² > threshold), it is marked as "complex" and needs to be further segmented. At this time, the current block is divided into 4 sub-blocks, and the uniformity measure is repeatedly calculated until the uniformity measure indicates that the current image is uniform (e.g., variance σ² ≤ threshold or the block size reaches the minimum threshold (e.g., 4×4 pixels)).

[0043] Optionally, the target block images are encoded independently only according to a preset encoding format, including: each target block image of the same video frame is encoded independently in a separate thread according to the preset encoding format.

[0044] In other words, it supports encoding target image blocks at the same location within the same thread, allowing simultaneous encoding of target image blocks at different locations. Compared to single-threaded encoding, this improves encoding speed and ensures video transmission rate. Furthermore, supporting independent encoding of image blocks at different locations reduces noise generated by mixing images from different locations, ensuring encoding accuracy.

[0045] In one example, the video frames are divided into x blocks, meaning the video display interface is divided into x independent windows. The blocks within each independent window are independent of each other. Within each independent window, there are steps such as similarity calculation, determining the target block based on similarity comparison, encoding the target block, and sending it. In this case, the video has x independent encoding threads, with different encoding threads corresponding to different independent windows. That is, an independent window (the target block image in the same position across all video frames) uses the same encoding thread for encoding. If a non-target block image exists within that independent window, it is skipped, and the next target block image in the window is encoded.

[0046] For example, the video display interface can be divided into x independent windows, which can be achieved by directly defining the video display interface, such as including the following instructions: typedef struct SpiceMonitor { uint32_t window_id; / / Used to define the window identifier for each window uint32_t x, y; / / Used to define the position coordinates of each window. These coordinates can be uniformly set to the position coordinates of the top right or top left corner of the window. uint32_t width, height; / / Used to define the size specifications of each window SpiceMonitor The SpiceMonitor structure defines a SpiceMonitor parameter, which in this example defines parameters for one or more independent windows. Optionally, this definition function may also include the data source for the current window or all windows, the encoder instance corresponding to the independent window, etc.

[0047] It is understandable that the segmented windows are no longer associated with each other; each window only updates the video display content corresponding to its own window. For example, each window may not be bound to a specific thread; instead, a thread pool concept can be used, with task preemption implemented when encoding is needed. That is, the thread pool may also have only one encoding thread, using task preemption to implement the encoding of the target segmented image as described in this application. For example, when any window has a target segmented image that needs encoding, the encoding task is submitted to the thread pool's task queue, where it is processed by an idle thread. If all threads are busy, the encoding task waits in the queue for execution.

[0048] Alternatively, the structure of an encoding task can be as follows: typedef struct SpiceEncodeTask { SpiceMonitor monitor; / / Defines the independent window corresponding to this encoding task, carrying the window identifier, position coordinates, size specifications, etc., which are the relevant parameters in the SpiceMonitor function defined above. Image encode_img; / / Defines the raw image data to be encoded for this encoding task. `encode_img` is a pointer that points to the specific data of the image to be encoded. struct SpiceEncodeTask next; / / Indicates that the current coding task will be passed to the next task. The `next` pointer is linked to the task queue of the aforementioned thread pool. SpiceEncodeTask; SpiceEncodeTask represents the current encoding task.

[0049] It's understandable that a larger number of threads in a thread pool can improve coding speed. A thread pool might have the following structure: pthread_t threads — Defines an array of threads; this is a pointer structure. threads can store the IDs of all coding threads, used for creating, managing, and recycling threads. int thread_num — Defines the size of the thread pool, that is, the number of worker threads in the thread pool (take twice the number of CPU cores, such as 8 threads, to make full use of CPU resources; or, in this example, it can be the same as the number of independent windows to ensure the encoding speed of each window). SpiceEncodeTask task_queue — Used to define the encoding task queue and points to the head of the encoding task queue. In addition, a mutex lock for the task queue can be added using the `pthread_mutex_t queue_mutex` instruction. Any operation on the coding task queue must acquire the lock first, preventing multiple coding threads from modifying the queue simultaneously and causing data inconsistency. Alternatively, a condition variable for the coding task queue can be set using the `pthread_cond_t queue_cond` instruction. For example, when the coding task queue is empty, the coding threads can be put into a waiting state, and when a task is added to the queue, the coding threads can be awakened. Alternatively, the `bool stop` instruction can be used to instruct the thread pool to stop. For example, when `stop=true`, the coding threads exit after processing all tasks, and the thread pool enters the shutdown process.

[0050] In one example, the steps for comparing the similarity of each window can be as follows: First, obtain the corresponding image block of the previous frame (e.g., `preImage`), then obtain the corresponding image block of the current frame (e.g., `currentImage`), and perform similarity calculation using a preset algorithm. In this example, the similarity is represented by the mean squared error between corresponding pixels in adjacent frames. The calculated similarity result is then compared with a preset threshold. If the similarity is low, it indicates a large change in the image, and encoding of this image block in the current frame is required. It's understandable that the mean squared error parameter indicates that the two parameters being calculated are closer when the value is smaller; therefore, when using the mean squared error value for similarity calculation, if the mean squared error value is greater than the preset mean squared error threshold, it means the similarity is less than the preset threshold.

[0051] For example, a separate window is located at the top right corner of a video frame, i.e. Figure 3 The positions of image blocks a2 and b2 in the video are determined by similarity comparison across five consecutive video frames. The image block at the top right corner of the third video frame is identical to the image block at the top right corner of the second video frame, meaning their similarity exceeds a preset threshold. However, the similarity between the top right corner images of other adjacent frames is less than the preset threshold. Therefore, the corresponding image block in the top right corner of this window in the third video frame is not a target image block and does not require processing. Thus, for the encoding thread of this window, only the image blocks at the top right corner of frames 1, 2, 4, and 5 need encoding. The independent encoding thread corresponding to this window continues to run (encoding the image blocks), skipping the image block at the top right corner of the third frame (i.e., the non-target image block) during the encoding process. During subsequent video decoding on the client side, the second frame image is used as the base image. Figure 3 The image in frame a, frame 3 is Figure 3Taking frame b as an example: Since there are no parameters in the upper right corner of the third frame (because there is no encoded transmission), position b2 in the third frame is blank. When the third frame is displayed on top of the second frame in the display order, position b2 is empty, which is just enough to display the block image at position a2. Therefore, when the third frame is displayed, b1, a2, b3, and b4 can be observed visually, that is, the image at the corresponding position (a2) of the second frame can be reused to achieve the complete display of the third video frame.

[0052] In this implementation, the multi-threaded approach also helps to distribute the encoding calculations of the virtual machine evenly across different CPU cores, thereby achieving CPU load balancing and reducing the situation where some virtual machines experience video stuttering while others experience smooth video playback.

[0053] Optionally, the preset encoding format is the MJPEG encoding format. MotionJoint Photographic Experts Group (MJPEG) is a video encoding format based on static JPEG image compression. It treats video as a series of independent JPEG image frames for compression and transmission, offering extreme simplicity, ultra-low latency, frame independence, and wide compatibility. When bandwidth and storage costs are not the primary bottleneck, and ease of implementation, device compatibility, processing speed, or frame accuracy are crucial, MJPEG is a reliable and practical choice. If bandwidth is sufficient, the MJPEG format can be used for video compression and transmission, significantly reducing CPU consumption compared to the mainstream H.264 format. MJPEG encodes each frame into a single JPEG image, meaning only intra-frame reference compression is used, without inter-frame compression, greatly reducing computational load.

[0054] It is understandable that, and since this solution is based on MJPEG for video encoding in practice, the encoding principle of MJPEG is to compress independent video frames according to the JPEG format. The basis of JPEG compression encoding is to use Discrete Cosine Transform (DCT) with an 8×8 block size for encoding. If an adaptive segmentation method based on image content is used when segmenting video frames, in addition to the large amount of computation, the shape and size of the segmented image blocks are not stable and cannot guarantee that the basic encoding specification (8×8 size) will be met. However, this application proposes to use a grid segmentation method, which results in segmented image blocks with fixed sizes and stable rectangular shapes, which can usually meet the encoding format.

[0055] Therefore, in some cases, the n×m block size used in the grid partitioning method is preferably a multiple of 8×8 required for DCT transformation to ensure that DCT transformation is completed smoothly.

[0056] Optionally, the independent encoding of the target segmented image according to a preset encoding format includes: encoding the target segmented image using the libjpeg-turbo library. It is understood that the ibjpeg-turbo encoding library can not only encode images according to the JPEG format, but also optimize the code through Single Instruction Multiple Data (SIMD), thereby improving the JPEG encoding and decoding speed. libjpeg-turbo is a high-performance JPEG image encoding and decoding library. The encoding process of the target segmented image includes, for example, reading the original images of each target segmented image, and independently compressing each target segmented image by calling jpeg_compress_struct. Here, jpeg_compress_struct is a structure in the libjpeg-turbo encoding library used to configure and execute JPEG image compression operations. Finally, the JPEG data of each target segmented image is saved (it can be stored separately or merged into a single file).

[0057] Optionally, the server is a Simple Protocol for Independent Computing Environments (SPICE) server, the video frames are obtained by the SPICE server from the virtual machine, and the client is a SPICE client. SPICE is an open-source remote desktop protocol primarily used for high-performance graphics transmission in virtualized environments (such as virtual machines). Its server is responsible for capturing, encoding, and transmitting screen changes from the virtual machine to the client.

[0058] The SPICE server performs operations such as segmentation, encoding, and transmission of video frames. During video processing, the SPICE server optimizes video transmission requirements, such as those for virtual desktops. For example, it supports a low-latency, high-frame-rate visual experience for multimedia applications; simultaneously, through intelligent compression, dedicated video stream optimization, and client-side rendering, it efficiently utilizes bandwidth and server resources. In other words, it effectively balances performance, image quality, bandwidth consumption, and server load.

[0059] In some implementations, video frames are segmented using a SPICE server. This can be achieved by calling the open-source interface of the original SPICE server and making some improvements to the original SPICE server, enabling the SPICE server to segment video frames.

[0060] In some examples, the server can also be a server that supports the Free Remote Desktop Protocol (FreeRDP), an open-source remote desktop protocol library that supports multiple remote desktop protocols, including the Remote Desktop Protocol (RDP). FreeRDP allows developers to implement RDP client and server functionality in their applications, enabling users to remotely access and control computers over a network. When the server is a FreeRDP-enabled server, the client is a FreeRDP-enabled client.

[0061] When using a FreeRDP-enabled server, the server-side processing procedure can refer to the processing method of the SPICE server described above. This server can use the Fast Forward Moving Picture Experts Group (FFmpeg) library for MJPEG encoding. It is understood that using the FFmpeg library for MJPEG encoding results in a higher compression ratio and requires less data transmission. However, it requires more CPU resources, thus placing certain demands on hardware configuration and resulting in higher hardware costs. Meanwhile, the FreeRDP-enabled server also supports multi-threaded operation, without limiting the number of working threads, which can improve data processing efficiency. If the libjpeg-turbo library can be used for MJPEG encoding on the FreeRDP-enabled server, then in some cases, this FreeRDP-enabled server can replace the aforementioned SPICE server to implement the video processing method provided in this application.

[0062] In some examples, the server can also be a server supporting Web Real-Time Communications (WebRTC), a real-time communication technology that allows web applications or websites to establish peer-to-peer connections between browsers without the need for an intermediary, enabling the transmission of video and / or audio streams or other arbitrary data. The standards included in WebRTC make it possible for users to create peer-to-peer data sharing and teleconferencing without installing any plugins or third-party software.

[0063] When using a WebRTC-enabled server, the server processing procedure can refer to the SPICE server processing method described above. This server commonly uses Google-developed VP8, VP9, ​​and AV1 encoding formats, but it also supports direct transmission of MJPEG format files. Specifically, MJPEG encoding can be implemented using FFmpeg or the open-source computer vision library (OpenCV).

[0064] This application provides a video processing method that divides video frames into blocks to obtain block images; compares the similarity of the block images with the previous video frame, determines the block images with lower similarity as target block images, and encodes only the target block images before sending them to the client. In other words, by reusing the encoding results of block images with higher similarity, the number of video frames or block images within video frames that need to be encoded is reduced, thereby reducing the demand for encoding and decoding calculations during video transmission and improving the speed of video encoding and decoding.

[0065] Specifically, in application, this means that only image blocks with low similarity are encoded and transmitted, while highly similar image blocks are not processed. Image blocks requiring encoding are encoded independently according to a preset encoding format. After the client decodes the acquired encoding results, because the video frame playback mode is that the subsequent frame overwrites the previous frame, when the current image block data is missing, the preceding image block at the same position in the previous frame will still be displayed in the missing space. This reuses the image block at the same position in the previous frame, thereby reducing the computational load for each video frame during encoding and decoding, and thus improving the speed of video processing.

[0066] In one optional embodiment of this application, the encoding result includes: the location information and temporal information corresponding to each of the target block images; then the method may further include: The system sends an encoded result carrying the location and timing information of the target segmented image to the client, allowing the client to decode the encoded result and display it. Specifically, when segmenting a video frame, the system obtains the location and timing information of the segmented image. The location information includes, for example, the coordinates of the segmented image within the video frame, and the timing information includes, for example, the specific time point in the target video when the segmented image should appear. The location and timing information are encoded together with the target segmented image to obtain an encoded result. The client decodes the encoded result to obtain a decoded result containing the target segmented image, its location information, and its timing information. The client then displays the corresponding target segmented image based on this location and timing information, thereby obtaining the target video.

[0067] In this embodiment, location information and timing information are also encoded and stored in the encoding result. For example, they can be placed as metadata in the data header of the encoding result of the target segment image, and the corresponding data body is the encoding result of the image data of the target segment image; or the location information and timing information are transmitted through channel a (such as a control channel), and the encoding result of the image data is transmitted through channel b (such as a display channel), that is, different parameters are transmitted to the client through two channels, so that the client can receive them and match them with the simplified location identifier carried in the image data in channel b and the complete location identifier and timing identifier in channel a to complete the final video display. Among them, the complete location identifier is such as the coordinate position of each boundary point of the target segment image and the length and width of the target segment image; the simplified location identifier is such as which segment the target segment image is in the video frame. Since it is usually transmitted in frames, the next frame is transmitted after the data in one video frame is transmitted. Therefore, in this method, the image data does not need to carry timing information, and the matching can be completed only by the simplified location identifier.

[0068] At this point, the coupling between the location and timing information and the image data encoding result of the target segment image is low. The location and timing information of the target segment image can be directly obtained by decoding the encoding result in the data header (or another transmission channel), without needing to fully decode the image data encoding result to obtain the location and timing information. This reduces the impact of image data on the location and timing information. Even when the image data encoding result is incorrect, the accurate location and timing information of the target segment image to be displayed can still be obtained, ensuring that the erroneous result only affects the display content of the current location and timing, without interfering with the display content of other locations and timing.

[0069] In this embodiment, the position information and timing information of the target segmented image are encoded together so that the client can arrange the target segmented image according to the position information and timing information, and finally obtain the target video.

[0070] Optionally, when the server is a SPICE server and the video frame is divided into x target block images, each encoded by x threads, the SPICE server divides the video into x independent windows for processing. The target block image is sent from the SPICE server to the SPICE client through the independent window corresponding to its position information. The position information of the target block image is obtained by the SPICE server and sent to the SPICE client at the initial stage of the independent encoding thread, and may not participate in the encoding / decoding process. Furthermore, before sending the target block image, the SPICE server can add timestamps / sequence numbers, etc., which will be used as the aforementioned timing information for the client to obtain the target video. That is, the aforementioned position information and timing information, along with the encoded image data, can be transmitted through different channels, and in this case, the position information and timing information can also be transmitted in their original format without encoding. Therefore, in some cases, timing information may not participate in the encoding / decoding process. After receiving the target block image, the SPICE client displays it at the correct position at the correct time node according to its corresponding independent window (or encoding thread).

[0071] Understandably, in some applications, when the server is a SPICE server, the location and timing information of the target image segments can be encoded and then decoded by a SPICE client to obtain the decoded location and timing information. This decoded location and timing information is used by the client to verify the correctness of the location and timing information obtained by the SPICE server through the aforementioned non-encoded path when obtaining the target video based on the decoding results, thus ensuring the accuracy of the target video acquisition.

[0072] For example, during the encoding process, the position information and timing information of the segmented images can be encoded and placed in the data header of the encoded result of the segmented images to be transmitted. At the same time, an encoding thread corresponding to the position information is used to encode the image, so that different windows can be encoded independently.

[0073] Position information includes, for example, the position of the top-left pixel of the current segmented image within the current video frame, as well as the length and width of the current segmented image, used to fully describe the absolute position of the current segmented image. For instance, if the current video frame size is 1920×1080, and n is 384 and m is 216 for the segmented image, the position information of the first segmented image in the first row from top to bottom is (0, 0, 384, 216), the position information of the second segmented image in the first row is (384, 0, 384, 216), the position information of the first segmented image in the second row is (0, 216, 384, 216), and so on.

[0074] Timing information can include, for example, the unique sequence number of the current video frame (e.g., an auto-incrementing natural number 1, 2, 3, etc.), indicating which frame the current segment of the image belongs to. Alternatively, timing information can also be the timestamp (in milliseconds or microseconds) of the current frame, used for audio-video synchronization, etc.

[0075] The window identifier for each independent window can be the specific location region of the current window within the video frame. Specifically, if the full size of the video frame is 1920×1080, and n is 384 and m is 216 for the segmented image, then the complete video frame can be divided into 25 segmented images, or 25 independent windows, arranged in 5 rows and 5 columns. For example, the identifier of the independent window in the first row and first column from top to bottom can be (0, 0) or 1, representing the first segment; the identifier of the independent window in the second row and second column can be (0, 1) or 2, representing the second segment, and so on.

[0076] Optionally, the number of encoding threads can be the same as the number of independent windows, ensuring that each window is assigned one encoding thread. This allows each window to complete encoding independently, preventing data pollution caused by different encoding threads processing data from different windows simultaneously. In this case, the thread ID of the encoding thread can be mapped to the window identifier. For example, after dividing the window into independent windows, obtaining the window identifiers, and obtaining the thread IDs in the thread pool, since the number is consistent, they can be randomly assigned, ensuring a one-to-one correspondence between windows and threads.

[0077] It is understandable that the location information, timing information, and window identifier mentioned above can be included in the header of the final encoded result as custom fields or encoded fields after simple encoding. When encoding is performed through the SPICE server, in some cases, the SPICE server will automatically encode the location information and timing information, without requiring specific command lines or other code segments to make the header of the encoded result carry the location or timing information.

[0078] In one optional embodiment of this application, obtaining the similarity between the current segmented image and the previous segmented image includes: S1, respectively obtain the parameter values ​​of the preset color channels of each pixel in the current block image and the forward block image; S2, calculate the mean square error between each pixel in the current block image and the forward block image based on the parameter values; S3, determine the similarity based on the mean square error.

[0079] That is, the similarity between the current block image and the previous block image is determined by the mean square error value between corresponding pixels. The calculation logic of the mean square error value is simple, the computing power requirement is small, the calculation result can be obtained quickly, and the hardware requirements are not increased.

[0080] Specifically, since the acquired video frames are in BGRX format, that is, each block image is also in BGRX format, there are red, green, blue and x channels. The x channel can be ignored in this calculation. Only the blue, green and red channels (B, G, R) are calculated for each corresponding pixel in the current block image and the corresponding forward block image.

[0081] First, obtain the parameter values ​​of the three color channels for each pixel, and calculate the difference between each color channel for pixels at the same position. Square the difference (to eliminate negative values ​​and amplify the difference). Then, sum the squared differences of all pixels and divide by the total number of pixels to obtain the Mean Squared Error (MSE). The smaller the MSE value, the more similar the images are. When the MSE calculated for the current block image and the previous block image is 0, it means that the two block images are completely identical, that is, the similarity is the highest, and the block image is not independently encoded.

[0082] Optionally, the MSE calculation part here can be optimized using the SIMD instruction set to further reduce the computational load. For example, optimization methods using the SIMD instruction set include: parallel data loading and calculation through vector registers; reducing the number of instructions: merging multiple operations into a single instruction; memory optimization: batch loading of data to reduce the number of memory accesses, etc.

[0083] Optionally, the similarity threshold used to determine the target image segments can be set based on empirical values. Specifically, a parameter value related to the video content or scene is set based on experience, without causing obvious segmentation effects or image fragmentation. For some high-motion scenes, the threshold through MSE can be set to 0, meaning the similarity threshold is completely identical, and any differences are encoded to meet the user's need to monitor motion details.

[0084] Understandably, calculating similarity using MSE, even with a threshold of 0 for MSE filtering, still reduces bandwidth and CPU computation due to the presence of identical image blocks. Furthermore, it allows for rapid acquisition of similarity scores and comparison results.

[0085] In this embodiment, the mean square error (MSE) is used to calculate the similarity between the current block image and the previous block image. This method requires less computing power and can quickly obtain the calculation results, thus improving the user experience.

[0086] like Figure 4As shown, this application also provides a video processing method applied to a client, the method comprising: Step 201: Receive the encoding result of the target block image sent by the server; the server divides each video frame into blocks to obtain block images; the target block image is determined by the server based on the similarity between the current block image and the preceding block image; the current block image is the block image in the current video frame, and the preceding block image is the block image in the previous video frame corresponding to the current block image; the encoding result is obtained by the server independently encoding the target block image according to a preset encoding format. Step 202: Decode the encoding result of the target block image according to the preset encoding format to obtain the target block image; Step 203: Obtain the target video based on the target segmented image; wherein, when there is a current segmented image with a similarity greater than a preset threshold, display the forward segmented image corresponding to the current segmented image.

[0087] In this embodiment, the server only sends the encoding result of the target segment image to the client. After receiving the encoding result of the target segment image sent by the server, the client can obtain the target segment image by decoding the encoding result. It can be understood that the target segment image is the current segment image whose similarity with the preceding segment image is less than a preset threshold. That is, the client only receives segment images with low similarity to the previous frame and obtains the target video based on this group of low-similarity segment images.

[0088] Specifically, the display process of video frames can involve one frame overwriting the previous one. When the client only receives the target segment image (i.e., only the segment image with low similarity to the previous frame), a complete video frame lacks a current segment image with a similarity greater than or equal to a preset threshold (i.e., a segment image with high similarity to the previous frame). Therefore, the complete video frame may contain segment images that have not been encoded and transmitted by the server, resulting in gaps in the video frame. In this case, since the playback or generation mode of the target video is one frame overwriting the previous frame, the missing part will display the image (segment image) at the corresponding position of the previous frame. It can be understood that the missing part was not selected as the target segment image for transmission because of its high similarity to the previous frame. Therefore, the image at the corresponding position of the previous frame displayed in the missing part completely matches the current video frame and does not affect the user's viewing. Thus, the client can generate the target video even by only receiving the target segment image, while reducing the computing power used by the client for video decoding and lowering hardware requirements.

[0089] That is, the client ultimately displays the target video based on the target segmented image; wherein, when displaying the current video frame, the corresponding forward segmented image is used to display the missing current segmented image (i.e., the segmented image whose similarity is greater than a preset threshold and therefore the server has not encoded and sent the segmented image).

[0090] Meanwhile, the client only needs to receive the encoding result of the target segmented image sent by the server. That is, both the server and the client only need to process the target segmented image. Segmented images with similarity higher than a preset threshold do not need to be processed. Therefore, the amount of computation for the server and the client is reduced, the computing power requirements of the server and the client are lowered, and the hardware cost of the server and client devices that load the server is reduced.

[0091] It is understandable that different frame rates are allowed at different positions of the target video during display, because the presence of each image block at a given location cannot be guaranteed in every frame.

[0092] For example, if a target image's original video displays 30 frames per second, and each frame is divided into four blocks using the same method: top left, top right, bottom left, and bottom right, then consecutive frames are divided into blocks like top left 1, top right 1, bottom left 1, bottom right 1; top left 2, top right 2, bottom left 2, bottom right 2, and so on. After similarity calculation, the similarity between top left 2, top left 3, and top left 1 is higher than a preset threshold; the similarity between top right 13 and top right 12 is higher than a preset threshold; the similarity between bottom left 24 and bottom left 24, and between bottom left 27, bottom left 28, and bottom left 26 is higher than a preset threshold; and the similarity of the bottom right block is lower than a preset threshold. Therefore, in 1 second, there are a total of 28 target block images in the upper left region, 29 target block images in the upper right region, 27 target block images in the lower left region, and 30 target block images in the lower right region; that is, during the display of the target video on the client, the frame rate may be different at different locations because the number of target block images at different locations may not be consistent.

[0093] In one optional embodiment of this application, the encoding result carries the location information and temporal information of the target block image; Then, the decoding of the encoding result of the target block image according to the preset encoding format includes: The encoding result is decoded according to the decoding format corresponding to the preset encoding format to obtain the position information and timing information corresponding to the target block image; Therefore, obtaining the target video based on the target segmented image includes: The target segmented images are stitched together based on their location and timing information to obtain the target video.

[0094] Specifically, when the server divides the video frame into blocks, it can obtain the position and timing information of the blocks. The position information identifies the coordinates of the block within the video frame, and the timing information is, for example, the time value at which the block should be displayed in the video, or the unique sequence number of the current video frame (e.g., an auto-incrementing natural number 1, 2, 3, etc.). After the client decodes the encoding result, it can obtain the target block image, its position and timing information, and correctly display the corresponding target block image at the corresponding position and time point based on the position and timing information, thus obtaining the target video.

[0095] For example, the position and timing information of the segmented image can be stored in the header of the encoded result. After the client decodes the encoded result, it can obtain the position and timing information corresponding to the current target segmented image from the header. Taking a full video frame size of 1920×1080 and a segmented image size of 384×216 as an example, if the position information of a segmented image is (0, 0, 384, 216) and the timing information is 2, as decoded from the header of the encoded result of a segmented image, it indicates that the segmented image is located in the first row and first column of the video frame, has a size of 384×216, and is displayed in the second frame.

[0096] In this embodiment, the server obtains the position and timing information of the target segmented image and encodes it simultaneously with the target segmented image. After decoding, the client arranges the target segmented image according to its position and timing information to obtain the target video. This reduces the impact of the encoded image data on the position and timing information. Specifically, the client does not need to decode the image data in the encoded result to obtain the position and timing information; it can directly obtain the position and timing information from a storage location independent of the encoded image data (such as a data header), and then arrange the decoded target segmented image. Even if the decoding operation of the encoded image data fails, because accurate position and timing information is obtained, it only affects the display of the current target segmented image and will not disrupt other video frame information due to data anomalies. This controls the scope of the impact of data transmission errors. Furthermore, the position and timing information obtained through data header decoding can accurately and quickly obtain the target video from the target segmented image, improving video display efficiency and user experience.

[0097] like Figure 5As shown in this embodiment, the server side includes a virtual machine and a SPICE server. The virtual machine provides the target video through its player and virtual graphics card. The target video is transmitted to the SPICE server as raw data (i.e., BGRX format). The SPICE server divides the video frames in the raw data into blocks. Inter-frame detection is performed on each block result (i.e., obtaining the similarity between the current block image and the preceding block image, and determining whether the current block image is the target block image based on the similarity compared to a preset threshold). The current block image is the block image in the current video frame, and the preceding block image is the block image in the previous video frame corresponding to the current block image. The target block image is determined based on the similarity. The current block image with a similarity less than the preset threshold is the target block image. The target block image is independently encoded according to a preset encoding format, and different encoding threads are used for different target block images to obtain the encoding result of the target block image. The encoding result is encapsulated and streamed, and sent to the client (cloud terminal) via the network for processing.

[0098] Figure 5 The diagram illustrates the encapsulation and streaming process after encoding (e.g., compression). This involves independently packaging the encoded results according to the application scenario (i.e., encapsulation) and sending them according to the network transmission protocol (i.e., streaming). For example, in a network surveillance camera scenario, the encoded results can be encapsulated into Real-time Transport Protocol (RTP) data packets, and then streamed using either RTP or Real-Time Streaming Protocol (RTSP). This combination offers advantages such as high image quality and low latency.

[0099] After receiving the encoding result of the target segmented image, the client decodes it according to the decoding thread corresponding to the encoding thread, and finally arranges the target segmented image according to the decoded data to obtain the target video and display it to the user.

[0100] In some implementations, reducing the transmission of one image block per second can cover the CPU consumption used for inter-frame similarity comparison and also improve processing speed.

[0101] In summary, this application provides a video processing method that divides video frames into blocks to obtain block images; compares the similarity of the block images with the previous video frame, determines the block images with lower similarity as target block images, and encodes only the target block images before sending them to the client. That is, by reusing the encoding results of block images with higher similarity, the number of video frames or block images within video frames that need to be encoded is reduced, thereby reducing the demand for encoding and decoding calculations during video transmission and improving the speed of video encoding and decoding.

[0102] Specifically, in application, this means that only image blocks with low similarity are encoded and transmitted, while highly similar image blocks are not processed. Image blocks requiring encoding are encoded independently according to a preset encoding format. After the client decodes the acquired encoding results, because the video frame playback mode is that the subsequent frame overwrites the previous frame, when the current image block data is missing, the preceding image block at the same position in the previous frame will still be displayed in the missing space. This reuses the image block at the same position in the previous frame, thereby reducing the computational load for each video frame during encoding and decoding, and thus improving the speed of video processing.

[0103] Reference Figure 6 This diagram illustrates a structural block diagram of a video processing apparatus according to this application, applied to a server. The video processing apparatus may include: The segmentation module 301 is used to segment each video frame into blocks and obtain segmented images; wherein, there is a corresponding relationship between the segmented images with the same position in each video frame. Selection module 302 is used to determine a target block image based on the similarity between the current block image and the forward block image; wherein, the current block image is a block image in the current video frame, and the forward block image is a block image in the previous video frame that corresponds to the current block image; The encoding module 303 is used to independently encode the target block image according to a preset encoding format, obtain the encoding result of the target block image and send it to the client, so that when the target video generated by the client contains a current block image with a similarity greater than a preset threshold, the forward block image corresponding to the current block image is displayed.

[0104] Optionally, the segmentation module is specifically used to: segment each video frame according to a preset n×m specification, and obtain at least one segmented image of the n×m specification corresponding to each video frame; where n and m are positive integers.

[0105] Optionally, the preset encoding format is the MJPEG encoding format.

[0106] Optionally, the encoding module is specifically used to: independently encode each target block image of the same video frame according to the preset encoding format in an independent thread.

[0107] Optionally, the server is a SPICE server, the video frame is obtained by the SPICE server from the virtual machine, and the client is a SPICE client.

[0108] Optionally, the encoding result includes: the location information and temporal information corresponding to each of the target block images; Therefore, the video processing device further includes: The system sends an encoding result carrying the location and timing information of the target block image to the client, so that the client can decode the encoding result and display the decoded result.

[0109] Optionally, the video processing apparatus further includes: The parameter acquisition module is used to acquire the parameter values ​​of the preset color channels of each pixel in the current block image and the forward block image, respectively. The parameter calculation module is used to calculate the mean square error between each pixel in the current block image and the forward block image based on the parameter values. A similarity determination module is used to determine the similarity based on the mean square error.

[0110] This application provides a video processing method that divides video frames into blocks to obtain block images; compares the similarity of the block images with the previous video frame, determines the block images with lower similarity as target block images, and encodes only the target block images before sending them to the client. In other words, by reusing the encoding results of block images with higher similarity, the number of video frames or block images within video frames that need to be encoded is reduced, thereby reducing the demand for encoding and decoding calculations during video transmission and improving the speed of video encoding and decoding.

[0111] Specifically, in application, this means that only image blocks with low similarity are encoded and transmitted, while highly similar image blocks are not processed. Image blocks requiring encoding are encoded independently according to a preset encoding format. After the client decodes the acquired encoding results, because the video frame playback mode is that the subsequent frame overwrites the previous frame, when the current image block data is missing, the preceding image block at the same position in the previous frame will still be displayed in the missing space. This reuses the image block at the same position in the previous frame, thereby reducing the computational load for each video frame during encoding and decoding, and thus improving the speed of video processing.

[0112] Reference Figure 7This diagram illustrates a structural block diagram of a video processing apparatus according to this application, applied to a client. The video processing apparatus may include: The receiving module 401 is used to receive the encoding result of the target block image sent by the server; the server divides each video frame into blocks to obtain block images; the target block image is determined by the server based on the similarity between the current block image and the previous block image; the current block image is the block image in the current video frame, and the previous block image is the block image in the previous video frame corresponding to the current block image; the encoding result is obtained by the server independently encoding the target block image according to a preset encoding format. Decoding module 402 is used to decode the encoding result of the target block image according to the preset encoding format to obtain the target block image; The generation module 403 is used to obtain a target video based on the target segmented image; wherein, when there is a current segmented image with a similarity greater than a preset threshold, the forward segmented image corresponding to the current segmented image is displayed.

[0113] Optionally, the encoding result carries the position information and timing information corresponding to each of the target block images; the decoding module 402 is specifically used to: decode the encoding result according to the decoding format corresponding to the preset encoding format to obtain the position information and timing information corresponding to the target block image; The generation module 403 is specifically used to: stitch together the target segmented image according to the position information and timing information corresponding to the target segmented image to obtain the target video.

[0114] In this embodiment, the server only sends the encoding result of the target segment image to the client. After receiving the encoding result of the target segment image sent by the server, the client can obtain the target segment image by decoding the encoding result. It can be understood that the target segment image is the current segment image whose similarity with the preceding segment image is less than a preset threshold. That is, the client only receives segment images with low similarity to the previous frame and obtains the target video based on this group of low-similarity segment images.

[0115] Specifically, the display process of video frames can involve one frame overwriting the previous one. When the client only receives the target segment image (i.e., only the segment image with low similarity to the previous frame), a complete video frame may lack a current segment image with a similarity greater than or equal to a preset threshold (i.e., a segment image with high similarity to the previous frame). Therefore, there may be gaps in the complete video frame. In this case, since the playback or generation mode of the target video is one frame overwriting the previous one, the missing part will display the image (segment image) at the corresponding position of the previous frame. It can be understood that the missing part was not selected as the target segment image for transmission because of its high similarity to the previous frame. Therefore, the image at the corresponding position of the previous frame displayed in the missing part completely matches the current video frame and does not affect the user's viewing. Thus, the client can generate the target video even by only receiving the target segment image, which can reduce the demand for encoding and decoding calculations during video transmission and thus improve the speed of video encoding and decoding.

[0116] Meanwhile, the client only needs to receive the encoding result of the target block image sent by the server. That is, both the server and the client only need to process the target block image, thus reducing the amount of computation for both the server and the client. This reduces the amount of computation for each video frame during the encoding and decoding process, thereby improving the speed of video processing.

[0117] Reference Figure 8 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the steps of the video processing method of the aforementioned embodiment.

[0118] This application provides a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a terminal's program or processor, enables the terminal to perform the steps of the video processing method described in the foregoing embodiments. This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video processing method, characterized in that, Applied to the server side, the method includes: Each video frame is divided into blocks to obtain block images; wherein, there is a correspondence between block images with the same position in each video frame; The target block image is determined based on the similarity between the current block image and the forward block image; wherein, the current block image is the block image in the current video frame, and the forward block image is the block image in the previous video frame that corresponds to the current block image; The target segment image is independently encoded according to a preset encoding format to obtain the encoding result of the target segment image and send it to the client. When the target video generated by the client contains a current segment image with a similarity greater than a preset threshold, the forward segment image corresponding to the current segment image is displayed.

2. The video processing method according to claim 1, characterized in that, The step of independently encoding only the target block image according to a preset encoding format includes: Each target block image of the same video frame is independently encoded in a separate thread according to the preset encoding format.

3. The video processing method according to claim 1, characterized in that, The encoding result includes: the location information and temporal information corresponding to each of the target block images; The method further includes: sending an encoding result carrying the location information and timing information of the target block image to the client, so that the client can decode the encoding result and display the decoded result.

4. The video processing method according to claim 1, characterized in that, The method further includes: Obtain the parameter values ​​of the preset color channels of each pixel in the current block image and the forward block image respectively; Calculate the mean square error between each pixel in the current block image and the forward block image based on the parameter values; The similarity is determined based on the mean square error.

5. The video processing method according to any one of claims 1 to 4, characterized in that, The step of dividing each video frame into blocks and obtaining block images includes: Each video frame is divided into blocks according to a preset n×m specification, and at least one block image of the n×m specification corresponding to each video frame is obtained; where n and m are positive integers.

6. A video processing method, characterized in that, Applied to a client, the method includes: The server receives the encoding result of the target block image sent by the server; the server divides each video frame into blocks to obtain block images; the target block image is determined by the server based on the similarity between the current block image and the previous block image; the current block image is the block image in the current video frame, and the previous block image is the block image in the previous video frame that corresponds to the current block image; the encoding result is obtained by the server independently encoding the target block image according to a preset encoding format. The encoding result of the target block image is decoded according to the preset encoding format to obtain the target block image; The target video is obtained based on the target segmented image; wherein, when there is a current segmented image with a similarity greater than a preset threshold, the forward segmented image corresponding to the current segmented image is displayed.

7. The video processing method according to claim 6, characterized in that, The encoding result carries the location information and temporal information corresponding to each of the target block images; Then, the decoding of the encoding result of the target block image according to the preset encoding format includes: The encoding result is decoded according to the decoding format corresponding to the preset encoding format to obtain the position information and timing information corresponding to the target block image; Therefore, obtaining the target video based on the target segmented image includes: The target segmented images are stitched together based on their location and timing information to obtain the target video.

8. A video processing apparatus, characterized in that, Applied to the server side, the device includes: The segmentation module is used to segment each video frame into blocks and obtain segmented images; wherein, there is a corresponding relationship between segmented images with the same position in each video frame. The selection module is used to determine the target block image based on the similarity between the current block image and the previous block image; wherein, the current block image is the block image in the current video frame, and the previous block image is the block image in the previous video frame that corresponds to the current block image; The encoding module is used to independently encode the target segment image according to a preset encoding format, obtain the encoding result of the target segment image and send it to the client, so that when the target video generated by the client contains a current segment image with a similarity greater than a preset threshold, the forward segment image corresponding to the current segment image is displayed.

9. A video processing apparatus, characterized in that, Applied to a client, the video processing device includes: A receiving module is used to receive the encoding result of the target block image sent by the server; the server divides each video frame into blocks to obtain block images; the target block image is determined by the server based on the similarity between the current block image and the preceding block image; the current block image is the block image in the current video frame, and the preceding block image is the block image in the previous video frame corresponding to the current block image; the encoding result is obtained by the server independently encoding the target block image according to a preset encoding format; A decoding module is used to decode the encoding result of the target block image according to the preset encoding format to obtain the target block image; The generation module is used to obtain a target video based on the target segmented image; wherein, when there is a current segmented image with a similarity greater than a preset threshold, the forward segmented image corresponding to the current segmented image is displayed.

10. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the steps of the video processing method as described in any one of claims 1 to 5 or 6 to 7.

11. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the video processing method as described in any one of claims 1 to 5 or 6 to 7.

Images