Video coding method and device and electronic equipment

Abstract

The invention provides a video coding method and device and electronic equipment, and relates to the technical field of video coding. The method comprises the following steps: in the process of carrying out partition coding on a target video sequence, if a motion region of the target video sequence is judged to change, switching a coding mode from a partition coding mode to a global coding mode; after the motion feature partition corresponding to the first video sequence is obtained, the coding mode is switched from the global coding mode to a partition coding mode; the first video sequence is a video sequence segment after first time in the target video sequence, and the first time is time for judging that a motion area of the target video sequence changes; and according to the motion feature partition, performing partition coding on the video sequence which is not coded in the target video sequence. According to the scheme of the invention, the coding partition of the video sequence can be adjusted in time, and the problem of partition coding failure caused by coding partition errors is avoided.

Description

Technical Field

[0001] The present invention relates to the field of video encoding technology, and in particular to a video encoding method, device and electronic equipment. Background Art

[0002] Video partition coding algorithm is a video coding technology that aims to improve coding efficiency and compression ratio by dividing the video image frame into multiple regions. Existing video partition coding algorithms mainly include three types: coding algorithms based on traditional block coding, object partitioning, and human visual characteristics.

[0003] In image-intensive application scenarios such as movies, games, etc., the picture changes of video image frames fluctuate greatly. The existing video partition coding algorithm divides the video image frame into fixed-size blocks and encodes and decodes each block independently. It is impossible to implement real-time partition coding adjustment for highly changing video data, which can easily lead to partition coding errors and make partition coding invalid. For example, in video coding, if a coding partition error occurs when processing a certain video image frame, the image information of certain areas may not be encoded, resulting in mosaic or missing areas in the area after decoding. Summary of the invention

[0004] The purpose of the present invention is to provide a video encoding method, device and electronic device, which solves the technical problem that the existing video encoding algorithm cannot perform partition encoding adjustment on highly changing video data, which easily leads to partition errors and makes the partition encoding invalid.

[0005] To achieve the above object, an embodiment of the present invention provides a video encoding method, comprising:

[0006] In the process of performing partition coding on the target video sequence, if it is determined that the motion region of the target video sequence changes, the coding mode is switched from the partition coding mode to the global coding mode;

[0007] After obtaining the motion feature partition corresponding to the first video sequence, the encoding mode is switched from the global encoding mode to the partition encoding mode; wherein the first video sequence is a video sequence segment located after a first time in the target video sequence, and the first time is a time when it is determined that the motion region of the target video sequence changes;

[0008] According to the motion feature partitions, a video sequence that has not been encoded in the target video sequence is partitioned and encoded.

[0009] Optionally, if it is determined that the motion area of ​​the target video sequence changes, the encoding mode is switched from the partition encoding mode to the global encoding mode, including:

[0010] Monitoring, by a first thread, whether a motion area of ​​the target video sequence changes;

[0011] When it is determined that the motion region changes, sending a partition adjustment notification to a second thread through the first thread, where the second thread is used to encode the target video sequence;

[0012] When receiving the partition adjustment notification, the second thread switches the encoding mode from the partition encoding mode to the global encoding mode.

[0013] Optionally, after acquiring the motion feature partition corresponding to the first video sequence, switching the encoding mode from the global encoding mode to the partition encoding mode includes:

[0014] Determine, by the first thread, a motion feature partition corresponding to the first video sequence, and send the motion feature partition to the second thread;

[0015] After receiving the motion feature partition, the second thread switches the encoding mode from the global encoding mode to the partition encoding mode.

[0016] Optionally, the step of obtaining motion feature partitions corresponding to the first video sequence includes:

[0017] Inputting the first video sequence into a region change optical flow estimation model, and outputting an optical flow estimation result;

[0018] Using an eight-neighborhood region partitioning algorithm, based on the optical flow estimation result, to obtain a motion region corresponding to the first video sequence;

[0019] The motion feature partition is obtained according to the change intensity of the motion area.

[0020] Optionally, before inputting the first video sequence into the region change optical flow estimation model, the method further includes:

[0021] The first video sequence is scaled according to a preset scaling factor.

[0022] Optionally, the motion feature partition includes: a first motion area and a second motion area, and the change intensity of the first motion area is greater than the change intensity of the second motion area.

[0023] Optionally, obtaining the motion feature partition according to the change intensity of the motion area includes:

[0024] In the first video sequence, if the average jump times of all pixels in the motion region is greater than or equal to the first times threshold, the motion region is determined to be the first motion area;

[0025] In the first video sequence, if the average jump times of all pixels in the motion area are less than the first times threshold, the motion area is determined to be the second motion area.

[0026] To achieve the above object, an embodiment of the present invention provides a video encoding device, comprising:

[0027] A first processing module is used for switching the encoding mode from the partition encoding mode to the global encoding mode if it is determined that the motion region of the target video sequence changes during the partition encoding of the target video sequence;

[0028] A second processing module is used to switch the encoding mode from the global encoding mode to the partition encoding mode after acquiring the motion feature partition corresponding to the first video sequence; wherein the first video sequence is a video sequence segment located after a first time in the target video sequence, and the first time is a time when it is determined that the motion region of the target video sequence changes;

[0029] The third processing module is used to perform partition coding on the uncoded video sequence in the target video sequence according to the motion feature partition.

[0030] To achieve the above object, an embodiment of the present invention provides an electronic device, including a processor and a transceiver, wherein the processor is used to:

[0031] In the process of performing partition coding on the target video sequence, if it is determined that the motion region of the target video sequence changes, the coding mode is switched from the partition coding mode to the global coding mode;

[0032] After obtaining the motion feature partition corresponding to the first video sequence, the encoding mode is switched from the global encoding mode to the partition encoding mode; wherein the first video sequence is a video sequence segment located after a first time in the target video sequence, and the first time is a time when it is determined that the motion region of the target video sequence changes;

[0033] According to the motion feature partitions, a video sequence that has not been encoded in the target video sequence is partitioned and encoded.

[0034] To achieve the above-mentioned purpose, an embodiment of the present invention provides an electronic device, comprising: a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the video encoding method is implemented.

[0035] To achieve the above objective, an embodiment of the present invention provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the above video encoding method are implemented.

[0036] To achieve the above objective, an embodiment of the present invention provides a computer program product, including computer instructions, which implement the steps in the above video encoding method when executed by a processor.

[0037] The beneficial effects of the above technical solution of the present invention are as follows:

[0038] In the above scheme, during the process of partition encoding of the target video sequence, if it is determined that the motion area of ​​the target video sequence has changed, the encoding method is switched from the partition encoding method to the global encoding method. By using global encoding to transition the partition scheduling process, the delay problem caused by untimely partitioning can be avoided. Further, if it is determined that the motion area of ​​the target video sequence has changed at the first time, a video sequence segment located after the first time is obtained as the first video sequence. After the motion feature partition corresponding to the first video sequence is obtained, the encoding method is switched from the global encoding method to the partition encoding method, and the video sequence that has not been encoded in the target video sequence is partitioned according to the motion feature partition. In this way, by preprocessing the video sequence segment with a changed motion area in the target video sequence to obtain the motion feature partition, the motion feature partition used for partition encoding can be updated in time, and the partition encoding is performed according to the obtained motion feature partition, which can avoid the problem of continuous partitioning of unchanged scenes, and timely adjust the encoding partition of the video sequence, and avoid the problem of invalid partition encoding due to encoding partition errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of a video encoding method according to an embodiment of the present invention;

[0040] Figure 2 This is one of the processing flow charts of video encoding according to an embodiment of the present invention;

[0041] Figure 3 FIG2 is a second processing flow chart of video encoding according to an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of motion feature partitioning according to an embodiment of the present invention;

[0043] Figure 5 is a structural diagram of a video encoding device according to an embodiment of the present invention;

[0044] Figure 6 is a structural diagram of an electronic device according to an embodiment of the present invention;

[0045] Figure 7 FIG. 4 is a structural diagram of an electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0047] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0048] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0049] like Figure 1 As shown, a video encoding method according to an embodiment of the present invention comprises the following steps:

[0050] Step 101: During the process of partition coding of a target video sequence, if it is determined that a motion region of the target video sequence changes, the coding mode is switched from the partition coding mode to the global coding mode.

[0051] Among them, global coding refers to using a unified coding method for the entire video image frame; partition coding refers to dividing the video image frame into multiple areas, and using a different coding method for each area.

[0052] It should be noted that the initial coding partition of the target video sequence is determined based on the first video sequence of the target video sequence, wherein the first video sequence is of a preset length; before obtaining the initial coding partition, the target video sequence is encoded using a global coding method. For example, for a video sequence of 100 frames, the first video sequence is 5 frames.

[0053] For example, Figure 2In the method, the target video sequence is globally encoded at time t0, and the initial encoding partition is determined according to the first video sequence; after the initial encoding partition is obtained at time t1, the partition encoding is switched to the partition encoding through the partition scheduling, and the target video sequence is partition encoded according to the initial encoding partition; further, in the process of partition encoding the target video sequence according to the initial encoding partition, if it is determined at time t2 that the motion area of ​​the target video sequence changes, the encoding mode is switched from the partition encoding mode to the global encoding mode.

[0054] In specific implementation, the optical flow changes of the two previous and next video frames can be monitored to determine whether the motion area of ​​the target video sequence has changed. For example, if the optical flow offset of the two previous and next video frames is greater than or equal to a preset threshold, it is determined that the motion area of ​​the target video sequence has changed.

[0055] Optical flow refers to the pattern of pixel motion between consecutive video frames, which can be used to estimate the motion of objects, the motion of the camera, or a combination of the two.

[0056] In this step, during the process of partition encoding of the target video sequence, if it is determined that the motion area of ​​the target video sequence has changed, the encoding method is switched from the partition encoding method to the global encoding method. This enables the use of global encoding to transition the partition scheduling process and avoid delay problems caused by untimely partitioning.

[0057] Step 102: After acquiring the motion feature partition corresponding to the first video sequence, the encoding method is switched from the global encoding method to the partition encoding method; wherein, the first video sequence is a video sequence segment in the target video sequence located after a first time, and the first time is the time for judging that the motion area of ​​the target video sequence changes.

[0058] In this step, if it is determined at the first time that the motion area of ​​the target video sequence has changed, a video sequence segment located after the first time is obtained as the first video sequence, and the motion feature partition corresponding to the first video sequence is obtained; after obtaining the motion feature partition corresponding to the first video sequence, the encoding method is switched from the global encoding method to the partition encoding method.

[0059] Optionally, the first video sequence includes a motion region and a non-motion region, and based on the change intensity of the motion region, the motion region may include at least two motion feature partitions. For example, the motion region includes a first motion region and a second motion region, a total of two motion feature partitions, and the change intensity of the first motion region is greater than that of the second motion region.

[0060] Optionally, the first frame image of the first video sequence may be a frame image corresponding to the first time, or the first frame image of the first video sequence may be a frame image that is located after the first time and is separated from the first time by a preset time length.

[0061] Optionally, the length of the first video sequence is a preset value. For example, the length of the first video sequence is 5 frames, 10 frames, 18 frames, 30 frames, 40 frames, 50 frames, etc., which is only an example and is not limited thereto.

[0062] Step 103: partition encoding of the unencoded video sequence in the target video sequence according to the motion feature partitions.

[0063] In this step, motion feature partitions are obtained by preprocessing the video sequence segments with motion area changes in the target video sequence. The motion feature partitions used for partition coding can be updated in time. Partition coding is performed based on the obtained motion feature partitions, which can avoid the problem of continuous partitioning of unchanged scenes.

[0064] For example, Figure 2 In the method, the target video sequence is globally encoded at time t0, and the initial encoding partition is determined according to the first video sequence; after the initial encoding partition is obtained at time t1, the partition encoding is switched to the partition encoding through the partition scheduling, and the target video sequence is partition encoded according to the initial encoding partition; further, in the process of partition encoding the target video sequence according to the initial encoding partition, if it is determined at time t2 that the motion area of ​​the target video sequence changes, the encoding mode is switched from the partition encoding mode to the global encoding mode; after the encoding mode is switched from the partition encoding mode to the global encoding mode, a video sequence segment after time t2 is selected as the first video sequence, and the motion feature partition corresponding to the first video sequence is re-acquired as the encoding partition basis; after the motion feature partition corresponding to the first video sequence is obtained at time t3, the global encoding mode is switched to the partition encoding mode, and the video sequence after time t3 is partition encoded according to the motion feature partition, until the next time the motion area is monitored to send a change, the above steps 101 to 103 are repeated; and so on, until the encoding is completed.

[0065] The above-mentioned embodiment can timely adjust the coding partition of the target video sequence, thereby avoiding the problem of partition coding failure due to coding partition error.

[0066] In some optional embodiments, if it is determined that the motion area of ​​the target video sequence changes, the encoding mode is switched from the partition encoding mode to the global encoding mode, including:

[0067] Monitoring, by a first thread, whether a motion area of ​​the target video sequence changes;

[0068] When it is determined that the motion region changes, sending a partition adjustment notification to a second thread through the first thread, where the second thread is used to encode the target video sequence;

[0069] When receiving the partition adjustment notification, the second thread switches the encoding mode from the partition encoding mode to the global encoding mode.

[0070] For example, Figure 3 In the example, the first thread monitors the change of the motion area of ​​the target video sequence at time t2, and sends a partition adjustment notification to the second thread; after receiving the partition adjustment notification, the second thread switches the encoding mode from the partition encoding mode to the global encoding mode.

[0071] In the above embodiment, the first thread monitors whether the motion area of ​​the target video sequence changes. If the motion area changes, the second thread is notified to adjust the encoding method, thereby realizing real-time adjustment of the encoding partitions. Moreover, through the cooperation of the first thread and the second thread, the encoding partition control in the encoding flow can be decoupled from the encoding process, thereby greatly reducing the amount of calculation increased by the change of the motion area in the entire encoding process, thereby ensuring the real-time and high efficiency of the encoding algorithm.

[0072] In some optional embodiments, after acquiring the motion feature partition corresponding to the first video sequence, the encoding mode is switched from the global encoding mode to the partition encoding mode, including:

[0073] Determine, by the first thread, a motion feature partition corresponding to the first video sequence, and send the motion feature partition to the second thread;

[0074] After receiving the motion feature partition, the second thread switches the encoding mode from the global encoding mode to the partition encoding mode.

[0075] For example, Figure 3 In the case where a change in the motion area of ​​the target video sequence is detected at time t2, a video sequence located after time t2 is selected as the first video sequence by the first thread, and the first video sequence is analyzed to obtain the motion feature partitions of the first video sequence; further, the motion feature partitions are sent to the second thread by the first thread, so that the second thread partitions and encodes the subsequent video sequences according to the second motion feature partitions.

[0076] In the above embodiment, the encoding partition process and the encoding partition adjustment function in the encoding process are decoupled through dual threads, which greatly reduces the amount of calculation increased by calculating the encoding partitions in the entire encoding process, thereby ensuring the real-time and high efficiency of the encoding algorithm.

[0077] It should be noted that the above embodiment divides the video encoding process into a first thread and a second thread, wherein the first thread is used to monitor the changes in the motion region of the entire target video sequence, and promptly notify the second thread to adjust the encoding method and encoding partition when the motion region changes. In this way, by using the mutual supervision between the two threads, the encoding partitions in the encoding process are smoothly and efficiently converted in a timely manner.

[0078] Exemplarily, the process of performing video encoding on a target video sequence using dual threads may include the following steps:

[0079] Step 1: Start the second thread to perform full-area encoding, and start the first thread to analyze the first video sequence of the target video sequence to obtain the initial encoding partition.

[0080] Step 2: The second thread performs partition coding according to the initial coding partition obtained by the first thread, and monitors whether a partition adjustment notification sent by the first thread is received.

[0081] Step 3: If the first thread identifies that the motion region of the target video sequence changes, it immediately sends a partition adjustment notification to the second thread to instruct the second thread to convert the encoding mode from partition encoding to global encoding;

[0082] Step 4: The first thread obtains the first video sequence segment as a basis for the next partitioning, and sends the obtained motion feature partition corresponding to the first video sequence to the second thread.

[0083] Step 5: After receiving the motion feature partitions sent by the first thread, the second thread switches the encoding mode from global encoding to partition encoding, and performs partition encoding on the unencoded video frames in the target video sequence according to the motion feature partitions.

[0084] For example, Figure 3As shown, at time t0, the target video sequence is globally encoded, and the initial encoding partition is determined according to the first video sequence; after the initial encoding partition is obtained at time t1, the first thread notifies the second thread to switch to partition encoding through partition scheduling; the second thread performs partition encoding on the target video sequence according to the initial encoding partition; further, in the process of the second thread performing partition encoding on the target video sequence according to the initial encoding partition, if the first thread determines at time t2 that the motion area of ​​the target video sequence has changed, the first thread sends a partition adjustment notification to the second thread, and the second thread switches the encoding mode from the partition encoding mode to the global encoding mode according to the partition adjustment notification; after the second thread switches the encoding mode from the partition encoding mode to the global encoding mode, the first thread selects a video sequence segment after time t2 as the first video sequence, and obtains the motion feature partition corresponding to the first video sequence; the first thread notifies the second thread to perform partition encoding on the target video sequence according to the initial encoding partition. The thread sends the motion feature partition corresponding to the first video sequence; after the second thread obtains the motion feature partition corresponding to the first video sequence at time t3, the encoding mode is switched from the global encoding mode to the partition encoding mode, and the video sequence after time t3 is encoded using the partition encoding mode, until the first thread monitors the change in the motion area at time t4, the first thread reselects a video sequence after time t4 as the first video sequence, and re-obtains the motion feature partition corresponding to the first video sequence; after the second thread receives the motion feature partition corresponding to the first video sequence sent by the first thread at time t5, the encoding mode is switched from the global encoding mode to the partition encoding mode, and the video sequence after time t5 is partition encoded according to the latest motion feature partition, until the next change in the motion area is monitored, and then the above steps 101 to 103 are repeated; and so on, until the encoding is completed.

[0085] In the above embodiment, when the second thread is started, it first performs global encoding and decoding operations on the entire captured image until it receives the initial encoding partition sent by the first thread; before sending the motion feature partition, the first thread analyzes the first segment of the captured video sequence, obtains the initial encoding partition, and sends the initial encoding partition to the second thread; further, during the partition encoding process, the first thread monitors the target video sequence in real time, and if the motion area of ​​the received video image frame changes, the second thread is notified in time, so that the second thread cancels the partition encoding and switches to global encoding, until the second thread receives the motion feature partition sent by the first thread, and then switches to partition encoding. In this way, through two threads, the partition adjustment process in the video encoding process can be separated from the encoding process, which greatly reduces the amount of calculation in the partition encoding process, and completes the intelligent encoding partition process by timely responding to the changes in the motion area in the target video sequence.

[0086] In some optional embodiments, the step of obtaining the motion feature partition corresponding to the first video sequence includes:

[0087] Inputting the first video sequence into a region change optical flow estimation model, and outputting an optical flow estimation result;

[0088] Using an eight-neighborhood region partitioning algorithm, based on the optical flow estimation result, to obtain a motion region corresponding to the first video sequence;

[0089] The motion feature partition is obtained according to the change intensity of the motion area.

[0090] The motion feature partitions include: a first motion area and a second motion area, and the change intensity of the first motion area is greater than the change intensity of the second motion area.

[0091] For example, Figure 4 In the embodiment, the motion area in the video picture includes the first motion area C and the second motion area A, and the non-motion area in the video picture is the area B except the first motion area C and the second motion area A.

[0092] It should be pointed out that the region change optical flow estimation model is a deep learning model for calculating optical flow, which can learn to estimate pixel motion from two consecutive image frames, thereby capturing dynamic information in the scene.

[0093] The following is an introduction to the regional change optical flow estimation model.

[0094] It should be noted that the regional variation optical flow estimation model (Spatial FlowNetwork, SFlownet) in the embodiment of the present application is obtained by optimizing the performance based on the network structure of FlownetC. ​​Specifically, the Flownet model is a deep learning model for calculating optical flow. The core of the Flownet model is a convolutional neural network (CNN), which is trained to predict the optical flow field between two consecutive image frames. There are two main variants of Flownet: FlownetS and FlownetC. ​​FlownetS is a direct end-to-end CNN that takes a pair of images as input and outputs an optical flow field. The network consists of a convolutional layer, a ReLU activation function, and a convolutional layer, and finally a combination of upsampling and convolutional layers to generate high-resolution optical flow predictions. FlownetC adds a correlation layer on the basis of FlownetS to calculate the degree of match between two feature maps, so that the network can better understand the motion relationship between image pairs. The training of the Flownet model usually involves the use of a large number of synthetic data sets that contain real optical flow fields for supervised learning. In this way, the network learns how to extract motion information from raw pixel data.

[0095] The main improvement of the regional change optical flow estimation model SFlownet in the embodiment of the present application is: integrating the attention mechanism (transformer) into FlownetC.

[0096] Specifically, the transformer module is used to receive the features extracted from the convolutional layer of FlownetC and enhance these features through the self-attention mechanism, so that the model can capture more global motion information in the image. In order to adapt to the processing method of the transformer module, the features need to be appropriately transformed in dimension and position encoding is added to retain spatial information; then, the features processed by the transformer module are fused with the original features to combine local and global information; then, the fused features are used to obtain the optical flow prediction results through a series of upsampling and convolution operations. After defining the loss function, the entire network is trained end-to-end and fine-tuned on a dataset with labeled optical flow; finally, the model is optimized by evaluating its performance on the validation set, and the trained model is used to identify new motion scenes to improve the accuracy and robustness of the model in complex motion scene recognition tasks.

[0097] In addition, the attention mechanism, especially the self-attention mechanism, allows the model to consider the relationship between all elements in the sequence when processing sequence data. In the context of image processing, the relationship between all pixels in the image can be considered simultaneously, not just the pixels in the local neighborhood. This global attention can help the model capture more complex spatial relationships and understand motion and dynamic scenes in the image.

[0098] In the above embodiment, by adding an attention mechanism to the region change optical flow estimation model (SFlownet), the model can be more focused on the small region change range and better focus on the key parts of the image, thereby extracting more effective features and improving the accuracy of the optical flow estimation results.

[0099] It should be pointed out that the optical flow estimation result is used to determine the moving area in the image. By analyzing the optical flow estimation result, the image can be divided into moving area and non-moving area. However, due to noise and distortion during data acquisition or data preprocessing, the optical flow change in a small area may be lost or not obvious, resulting in incomplete identification of the moving area. In order to improve the accuracy of moving area identification, the present application adopts an eight-neighborhood area partitioning algorithm to obtain the moving area corresponding to the first video sequence based on the optical flow estimation result.

[0100] Among them, the eight-neighborhood region partitioning algorithm uses the correlation between regions to traverse the eight neighborhoods that generate optical flow pixels, takes no optical flow change as the judgment condition, and traverses the image with the central optical flow pixel as the starting point until there is no optical flow change in the neighborhood.

[0101] Exemplarily, the eight-neighborhood region partitioning algorithm may include the following steps:

[0102] Step 1: Set a growth criterion: In the eight-neighborhood area of ​​the current pixel, if there are three adjacent pixels that are optical flow change pixels, they will be included in the change range, otherwise they will be excluded.

[0103] The eight-neighborhood area of ​​the current pixel includes: an upper pixel, an upper right pixel, a right pixel, a lower right pixel, a lower pixel, a lower left pixel, a left pixel, and an upper left pixel.

[0104] Step 2: Scan the image pixel sequence in order to find the first pixel that has not been assigned, and set the coordinates of the pixel to (x0, y0); take (x0, y0) as the center, consider the eight neighboring pixels (x, y) of (x0, y0), if the growth criterion is met, merge (x, y) with (x0, y0), and push (x0, y0) into the stack; after pushing, take out the next unassigned pixel, use it as the center pixel, and repeatedly determine whether the growth criterion is met until all motion areas are recorded in the stack, and then end the partitioning.

[0105] Step 3: Determine the motion area in the image based on the positions of the pixels entering the stack.

[0106] Specifically, by determining the position of the upper leftmost pixel and the lower rightmost pixel of each stacked pixel position, the rectangular motion area is determined, thereby obtaining the sizes of different motion area blocks, that is, the area of ​​motion change.

[0107] In the above embodiment, the eight-neighborhood region division algorithm is used to summarize the motion region according to the neighborhood optical flow change, and the change region in the optical flow estimation result is more accurately selected to achieve more accurate and efficient identification and division of the motion region; further, the motion feature partition is determined according to the change intensity of the motion region.

[0108] As an implementation method, in order to calculate the change intensity of the motion area, the average number of optical flow changes of pixels in the motion area is cumulatively calculated when dividing the motion area; based on the average number of optical flow changes, the change intensity of the motion area is determined.

[0109] As another implementation, in order to calculate the change intensity of the motion area, the average jump number of pixels in the motion area is accumulated when dividing the motion area; the change intensity of the motion area is determined based on the average jump number. In specific implementation, the change intensity can be determined based on the color similarity of the pixels in the motion area. For example, if the color of a pixel in a motion area in two adjacent frames of images is less than a preset threshold, it is determined that the pixel has jumped.

[0110] In some optional embodiments, obtaining the motion feature partition according to the change intensity of the motion area includes:

[0111] In the first video sequence, if the average jump times of all pixels in the motion region is greater than or equal to the first times threshold, the motion region is determined to be the first motion area;

[0112] In the first video sequence, if the average jump times of all pixels in the motion area are less than the first times threshold, the motion area is determined to be the second motion area.

[0113] The change intensity of the first motion area is greater than the change intensity of the second motion area.

[0114] For example, if a 60-frame video sequence is used as a unit, if the average jump number of all pixels in a certain motion area is less than or equal to 10 times, the motion area is classified as a weak change area, that is, the second area; if the average jump number of pixels in a certain motion area is greater than 10 times, the motion area is classified as a strong change area, that is, the first area.

[0115] In the above embodiment, based on the optical flow estimation result, the eight-neighborhood region division algorithm is combined to calculate the motion region area of ​​the entire picture data to obtain the motion region division. Furthermore, based on the change intensity of the motion region, the motion region is divided into a strong change region and a weak change region. In this way, the motion region change range is combined with the motion region change intensity to perform a more three-dimensional and comprehensive classification and description of the motion characteristics of the first video sequence, thereby improving the interpretability and robustness of the coding partition, enabling the partition encoding and decoding process to focus more on the motion characteristics of different motion regions, improving the speed and accuracy of motion region identification, and improving coding efficiency.

[0116] Exemplarily, in the partition coding process, no frame loss is adopted for the strong motion area, and the current bit rate value is maintained, because the strong motion area is mostly the focus of the user's current behavior, and the picture quality of this area will most directly affect the user's experience, so the encoding of the strong change area is not weakened. Secondly, since most of the weak change areas are non-key focus areas, such as bullet screen areas and advertising areas, in the partition coding process, the bit rate value of the weak change area is reduced according to the degree of optical flow change of the video sequence. Because the frame rate of the weak change area itself is upper bounded by 10fps / s, reducing the frame rate has little effect on improving the transmission efficiency, so no frame loss is performed. Finally, for the non-motion area, the image data in the first frame of the video is directly used without re-encoding or compression to reduce processing time and computing resource consumption.

[0117] In some optional embodiments, before inputting the first video sequence into the region variation optical flow estimation model, the method further includes:

[0118] The first video sequence is scaled according to a preset scaling factor.

[0119] For example, by reducing it by 3 times, the original data is compressed to 1 / 3 of the original size, and after the motion area division is completed, the image data is restored.

[0120] In the above embodiment, in order to more efficiently identify and divide the motion area and reduce time consumption, the collected image data is first scaled. This can minimize the impact of small area motion on the overall area division while ensuring that the motion characteristics of the large-scale motion area remain unchanged, reduce the computational complexity of the encoding process, and perform scene partitioning at the lowest cost.

[0121] The above scheme includes at least the following technical advantages:

[0122] 1. Meet the real-time and low-consumption requirements of the cloud computer transmission service process. A large amount of data needs to be transmitted in real time during the cloud computer service process, so the partition encoding process cannot occupy too many computing resources to avoid affecting the computing resources of other transmission channels of the cloud computer. In the existing partition encoding algorithm, the partition process will be directly affected by the performance of the image processing algorithm, and the process of batch processing of video data will consume a lot of computing resources, and real-time and low-consumption cannot be guaranteed. The video encoding scheme of the present application, by establishing a dual-thread management mechanism, performs segmented pre-processing on the collected video image sequence, avoids the situation of continuous partitioning of unchanged scenes, realizes intelligent adjustment of partitions, minimizes the amount of calculation caused by partitioning, and uses global coding to transition the partition process, avoiding the delay problem caused by untimely partitioning.

[0123] 2. Regional division based on motion patterns improves the accuracy and interpretability of regional division. In the process of cloud computer transmission services, the motion characteristics in cloud computer application scenarios are relatively single. The solution of this application introduces the change intensity of the motion area on the basis of considering the range of motion changes, and further describes the motion feature partitioning. Compared with the existing partition coding algorithm, it not only improves the partition efficiency, but also enhances the interpretability of the partition, which enables the encoding process to pay more attention to the motion changes in the current area.

[0124] 3. This application is based on the optical flow estimation network model. By scaling the input data, introducing the attention mechanism for prediction, and using the eight-neighborhood region partitioning algorithm to analyze and process the output optical flow estimation results, the final motion area is obtained, thereby improving the recognition speed and accuracy of the motion area.

[0125] 4. The solution of this application does not rely on the implementation of the operating system or hardware driver, and can be quickly and perfectly compatible with other operating systems and hardware, greatly reducing the workload of cross-platform development.

[0126] 5. The solution of this application can encode according to different areas of the image screen, and perform targeted compression according to the characteristics of each area, thereby significantly improving the compression rate and transmission efficiency of the data. This is particularly critical for large-scale data transmission or bandwidth-limited situations. For example, in application scenarios such as video streaming and network communication, it can increase transmission speed, reduce latency, and improve video quality, bringing users a smoother, high-quality cloud experience, thereby improving the performance of cloud computer products. It can also effectively reduce the bandwidth required for video transmission, thereby reducing operating costs.

[0127] like Figure 5 As shown, a video encoding device according to an embodiment of the present invention includes:

[0128] The first processing module 501 is used for switching the encoding mode from the partition encoding mode to the global encoding mode if it is determined that the motion region of the target video sequence changes during the partition encoding of the target video sequence;

[0129] The second processing module 502 is used to switch the encoding mode from the global encoding mode to the partition encoding mode after acquiring the motion feature partition corresponding to the first video sequence; wherein the first video sequence is a video sequence segment located after a first time in the target video sequence, and the first time is a time when it is determined that the motion region of the target video sequence changes;

[0130] The third processing module 503 is used to partition and encode the unencoded video sequence in the target video sequence according to the motion feature partitions.

[0131] The first processing module 501 is specifically used for:

[0132] Monitoring, by a first thread, whether a motion area of ​​the target video sequence changes;

[0133] When it is determined that the motion region changes, sending a partition adjustment notification to a second thread through the first thread, where the second thread is used to encode the target video sequence;

[0134] When the second thread receives the partition adjustment notification, the encoding mode is switched from the partition encoding mode to the global encoding mode.

[0135] Optionally, the second processing module 502 is specifically configured to:

[0136] Determine, by the first thread, a motion feature partition corresponding to the first video sequence, and send the motion feature partition to the second thread;

[0137] After receiving the motion feature partition, the second thread switches the encoding mode from the global encoding mode to the partition encoding mode.

[0138] Optionally, the second processing module 502 includes:

[0139] A first processing submodule, used for inputting the first video sequence into a region change optical flow estimation model, and outputting an optical flow estimation result;

[0140] A second processing submodule is configured to use an eight-neighborhood region partitioning algorithm to obtain a motion region corresponding to the first video sequence based on the optical flow estimation result;

[0141] The third processing submodule is used to obtain the motion feature partition according to the change intensity of the motion area.

[0142] Optionally, the apparatus 500 further includes:

[0143] The fourth processing module is used to perform scaling processing on the first video sequence according to a preset scaling factor.

[0144] Optionally, the motion feature partition includes: a first motion area and a second motion area, and the change intensity of the first motion area is greater than the change intensity of the second motion area.

[0145] Optionally, the third processing submodule includes:

[0146] A first processing unit is configured to determine, in the first video sequence, that the motion region is the first motion area if an average jump number of all pixels in the motion region is greater than or equal to the first number threshold;

[0147] The second processing unit is configured to determine, in the first video sequence, that the motion region is the second motion area if an average jump number of all pixels in the motion region is less than the first jump number threshold.

[0148] The video encoding device of this embodiment is a device corresponding to the above-mentioned video encoding method. All implementation means in the above-mentioned method embodiment are applicable to the embodiment of the video encoding device and can achieve the same technical effect.

[0149] like Figure 6 As shown, an electronic device 600 according to an embodiment of the present invention includes a processor 610 and a transceiver 620, wherein the processor 610 is used to switch the encoding mode from the partition encoding mode to the global encoding mode if it is determined that the motion region of the target video sequence changes during the partition encoding of the target video sequence;

[0150] The processor 610 is further configured to switch the encoding mode from the global encoding mode to the partition encoding mode after acquiring the motion feature partition corresponding to the first video sequence; wherein the first video sequence is a video sequence segment located after a first time in the target video sequence, and the first time is a time when it is determined that the motion region of the target video sequence changes;

[0151] The processor 610 is further configured to perform partition coding on a video sequence that has not been coded in the target video sequence according to the motion feature partitions.

[0152] Optionally, the processor 610 is specifically configured to:

[0153] Monitoring, by a first thread, whether a motion area of ​​the target video sequence changes;

[0154] When it is determined that the motion region changes, sending a partition adjustment notification to a second thread through the first thread, where the second thread is used to encode the target video sequence;

[0155] When receiving the partition adjustment notification, the second thread switches the encoding mode from the partition encoding mode to the global encoding mode.

[0156] Optionally, the processor 610 is specifically configured to:

[0157] Determine, by the first thread, a motion feature partition corresponding to the first video sequence, and send the motion feature partition to the second thread;

[0158] After receiving the motion feature partition, the second thread switches the encoding mode from the global encoding mode to the partition encoding mode.

[0159] Optionally, the processor 610 is specifically configured to:

[0160] Inputting the first video sequence into a region change optical flow estimation model, and outputting an optical flow estimation result;

[0161] Using an eight-neighborhood region partitioning algorithm, based on the optical flow estimation result, to obtain a motion region corresponding to the first video sequence;

[0162] The motion feature partition is obtained according to the change intensity of the motion area.

[0163] Optionally, the processor 610 is further configured to:

[0164] The first video sequence is scaled according to a preset scaling factor.

[0165] Optionally, the motion feature partition includes: a first motion area and a second motion area, and the change intensity of the first motion area is greater than the change intensity of the second motion area.

[0166] Optionally, the processor 610 is specifically configured to:

[0167] In the first video sequence, if the average jump times of all pixels in the motion region is greater than or equal to the first times threshold, the motion region is determined to be the first motion area;

[0168] In the first video sequence, if the average jump times of all pixels in the motion area are less than the first times threshold, the motion area is determined to be the second motion area.

[0169] The electronic device in this embodiment, during the process of partition encoding of the target video sequence, if it is determined that the motion region of the target video sequence has changed, the encoding mode is switched from the partition encoding mode to the global encoding mode; after obtaining the motion feature partition corresponding to the first video sequence, the encoding mode is switched from the global encoding mode to the partition encoding mode; wherein the first video sequence is a video sequence segment located after a first time in the target video sequence, and the first time is the time when the motion region of the target video sequence is determined to have changed; according to the motion feature partition, the video sequence that has not been encoded in the target video sequence is partition encoded. In this way, the encoding partition of the video sequence can be adjusted in a timely manner to avoid the problem of partition encoding failure due to encoding partition errors.

[0170] An electronic device according to another embodiment of the present invention, such as Figure 7 As shown, it includes a transceiver 710, a processor 700, a memory 720, and a program or instruction stored in the memory 720 and executable on the processor 700; when the processor 700 executes the program or instruction, the above-mentioned video encoding method is implemented.

[0171] The transceiver 710 is used to receive and send data under the control of the processor 700.

[0172] Among them, Figure 7In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 700 and various circuits of memory represented by memory 720 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 710 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 730 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0173] The processor 700 is responsible for managing the bus architecture and general processing, and the memory 720 can store data used by the processor 700 when performing operations.

[0174] A computer-readable storage medium according to an embodiment of the present invention stores a program or instruction thereon, and when the program or instruction is executed by a processor, the steps in the video encoding method described above are implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0175] It should be further explained that the terminals described in this specification include but are not limited to smart phones, tablet computers, etc., and many of the functional components described are called modules in order to more particularly emphasize the independence of their implementation methods.

[0176] In the embodiment of the present invention, module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.

[0177] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.

[0178] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.

[0179] The above exemplary embodiments are described with reference to the accompanying drawings, and many different forms and embodiments are feasible without departing from the spirit and teachings of the present invention. Therefore, the present invention should not be constructed as a limitation of the exemplary embodiments proposed herein. More specifically, these exemplary embodiments are provided so that the present invention will be perfect and complete, and the scope of the present invention will be conveyed to those who are familiar with the technology. In these figures, the component sizes and relative sizes may be exaggerated for clarity. The terms used here are only based on the purpose of describing specific exemplary embodiments and are not intended to be limiting. As used herein, unless the text clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to include these multiple forms. It will be further understood that the terms "including" and / or "comprising" when used in this specification indicate the presence of the features, integers, steps, operations, components and / or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, components, components and / or their groups. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of that range and any subranges therebetween.

[0180] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A video encoding method, characterized in that: include: In the process of performing partition coding on the target video sequence, if it is determined that the motion region of the target video sequence changes, the coding mode is switched from the partition coding mode to the global coding mode; After acquiring the motion feature partition corresponding to the first video sequence, the encoding mode is switched from the global encoding mode to the partition encoding mode; wherein the first video sequence is a video sequence segment located after a first time in the target video sequence, and the first time is a time when it is determined that the motion region of the target video sequence changes; According to the motion feature partitions, a video sequence that has not been encoded in the target video sequence is partitioned and encoded.

2. The video encoding method according to claim 1, characterized in that If it is determined that the motion area of ​​the target video sequence changes, the encoding mode is switched from the partition encoding mode to the global encoding mode, including: Monitoring, by a first thread, whether a motion area of ​​the target video sequence changes; When it is determined that the motion region changes, sending a partition adjustment notification to a second thread through the first thread, where the second thread is used to encode the target video sequence; When receiving the partition adjustment notification, the second thread switches the encoding mode from the partition encoding mode to the global encoding mode.

3. The video encoding method according to claim 1, characterized in that: After acquiring the motion feature partition corresponding to the first video sequence, the encoding mode is switched from the global encoding mode to the partition encoding mode, including: Determine, by a first thread, a motion feature partition corresponding to the first video sequence, and send the motion feature partition to a second thread; After receiving the motion feature partition, the second thread switches the encoding mode from the global encoding mode to the partition encoding mode.

4. The video encoding method according to claim 1, characterized in that: The step of obtaining motion feature partitions corresponding to the first video sequence includes: Inputting the first video sequence into a region change optical flow estimation model, and outputting an optical flow estimation result; Using an eight-neighborhood region partitioning algorithm, based on the optical flow estimation result, to obtain a motion region corresponding to the first video sequence; The motion feature partition is obtained according to the change intensity of the motion area.

5. The video encoding method according to claim 4, characterized in that: Before inputting the first video sequence into the region variation optical flow estimation model, the method further includes: The first video sequence is scaled according to a preset scaling factor.

6. The video encoding method according to claim 1 or 4, characterized in that: The motion feature partitions include: a first motion area and a second motion area, and the change intensity of the first motion area is greater than the change intensity of the second motion area.

7. The video encoding method according to claim 6, characterized in that: The step of obtaining the motion feature partition according to the change intensity of the motion area includes: In the first video sequence, if the average jump times of all pixels in the motion region is greater than or equal to a first number threshold, the motion region is determined to be the first motion area; In the first video sequence, if the average jump times of all pixels in the motion area are less than the first times threshold, the motion area is determined to be the second motion area.

8. A video encoding device, characterized in that: include: A first processing module is used for switching the encoding mode from the partition encoding mode to the global encoding mode if it is determined that the motion region of the target video sequence changes during the partition encoding of the target video sequence; A second processing module is configured to switch the encoding mode from the global encoding mode to the partition encoding mode after acquiring the motion feature partition corresponding to the first video sequence; wherein the first video sequence is a video sequence segment located after a first time in the target video sequence, and the first time is a time when it is determined that the motion region of the target video sequence changes; The third processing module is used to perform partition coding on the uncoded video sequence in the target video sequence according to the motion feature partition.

9. An electronic device, comprising: A transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; wherein the processor implements the video encoding method according to any one of claims 1 to 7 when executing the program or instruction.

10. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps in the video encoding method according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the video encoding method according to any one of claims 1 to 7.

Images