Neural network based inter-frame macroblock motion vector determination method and apparatus

Abstract

The application discloses a kind of interframe macroblock motion vector determination method and device based on neural network, it is related to video processing technical field, wherein, the method includes: using improved U2net neural network to carry out semantic segmentation to video frame, determine the foreground area and background area in video frame, based on reference object in foreground area or background area, using depth local feature search network analysis video key feature sequence before and after frame, and based on key feature sequence determines target search area, depth local feature search network has fused time series neural network based on bidirectional long short-term memory network, carries out encoder interframe macroblock motion vector analysis strategy in target search area, determines the macroblock motion vector in target search area.The present application solves the technical problem that in the related art, in the video frame encoding process, using local feature to carry out MV prediction easily leads to improper search range selection, which affects the prediction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing or other related fields, in particular, to a neural network-based inter-frame macroblock motion vector determination method and device. BACKGROUND

[0002] In modern video communication and storage systems, video coding plays a crucial role, and its goal is to effectively compress video data by removing redundant information in video signals, thereby reducing storage space and transmission bandwidth requirements. As an advanced video coding standard, H.265 / HEVC (High Efficiency Video Coding) provides more efficient data compression technology compared to the previous H.264 / AVC standard, especially in the encoding of high-resolution videos. Inter-prediction is a key technology in HEVC encoders, which utilizes the temporal and spatial redundancy between adjacent frames in a video sequence to predict the motion state of a macroblock in the current frame by finding the best matching block in the reference frame. The positioning of this best matching block is described by a motion vector (MV), which points to the pixel block in the reference frame that is most similar to the current macroblock. In HEVC, to improve prediction efficiency and reduce the bit rate of prediction information, Merge and AMVP (Advanced Motion Vector Prediction) prediction techniques are introduced. Both techniques provide one or more possible MV candidates for the current macroblock by analyzing the motion information of neighboring macroblocks, thereby narrowing the range of global search and reducing computational complexity. In the Merge mode, the MV of the current macroblock is directly inherited from the MV of its surrounding macroblocks, without the need for additional encoding of MV differences. This method assumes that the motion trend of adjacent macroblocks in a video sequence is similar at the same time, thereby achieving efficient prediction and coding of MV. The AMVP mode allows for fine-tuned adjustments to the estimated MV to obtain more accurate matching, although this slightly increases the bit rate of MV information.

[0003] However, the existing HEVC inter-frame prediction technology still has various disadvantages: 1. When the local feature in the video content is not obvious, especially in the scene with a single color space, the MV prediction using the local feature may cause improper selection of the search range, affecting the prediction accuracy. 2. In the fast moving scene, the traditional technology such as AMVP and Merge mode may not accurately predict the motion of the macroblock, especially when the macroblock moves across a large distance, the deviation in the selection of the search starting point may cause a high prediction error. 3. In the video encoding, the processing strategy for the background and foreground regions lacks differentiation, which often causes the background region to occupy too much code stream, and the encoding quality of the foreground region is limited, affecting the understanding and analysis of the video content.

[0004] Currently, there is no effective solution to the above problems. SUMMARY

[0005] The embodiment of the present application provides a neural network-based inter-frame macroblock motion vector determination method and device, so as to at least solve the technical problem that in the video frame encoding process in the related art, the MV prediction using the local feature is easy to cause improper selection of the search range, affecting the prediction accuracy.

[0006] In order to achieve the above purpose, according to one aspect of the present application, a neural network-based inter-frame macroblock motion vector determination method is provided, comprising: performing semantic segmentation on a video frame by using an improved U2net neural network, determining a foreground region and a background region in the video frame; based on a reference object in the foreground region or the background region, using a deep local feature search network to analyze the key feature sequence of the video front and rear frames, and determining a target search region based on the key feature sequence, wherein the deep local feature search network has fused a time sequence neural network based on a bidirectional long short-term memory network, and the key feature sequence includes feature points of a plurality of key objects and a preliminary motion vector, the key object refers to an object in the video front and rear frames with a similarity greater than a preset similarity threshold; performing an encoder inter-frame macroblock motion vector analysis strategy in the target search region to determine the macroblock motion vector in the target search region.

[0007] Optionally, the step of determining the foreground region and the background region in the video frame by using the improved U2net neural network includes: extracting features in the video frame by using an Encoder layer with a U structure to obtain a feature map set; performing a multi-scale dilated convolution operation on each feature map in the feature map set by using an all-sky pyramid pooling structure (ASPP) and performing an upsampling operation to obtain a feature map set with the same size; performing a convolution dimension reduction operation on a channel feature map in the feature map set with the same size to obtain a 2-dimensional vector matrix; performing channel fusion and normalization processing on elements in the 2-dimensional vector matrix to output a probability value, wherein the probability value is used to indicate whether there is a foreground object in the video frame; and in a case where the probability value indicates that there is a foreground object in the video frame, determining a region containing the foreground object as the foreground region and determining other regions except the foreground region as the background region.

[0008] Optionally, the step of performing a convolution dimension reduction operation on a channel feature map in the feature map set with the same size to obtain a 2-dimensional vector matrix includes: performing a nonlinear conversion processing on each channel feature map by using a one-dimensional convolution layer to obtain a converged 8-dimensional vector; and performing a convolution dimension reduction operation on the converged 8-dimensional vector based on a two-layer convolution loss function to obtain a 2-dimensional vector matrix, wherein the two-layer convolution loss function fuses a loss function of an Encoder layer with a U structure, an image texture feature loss function and an image overall feature loss function.

[0009] Optionally, after the step of performing channel fusion and normalization processing on elements in the 2-dimensional vector matrix to output a probability value, the method further includes: determining an image coordinate origin of the foreground region containing the foreground object; and constructing a coordinate system with the image coordinate origin as a base point to output a foreground region segmentation coordinate.

[0010] Optionally, the step of analyzing key feature sequences of the video front and back frames by using a deep local feature search network and determining a target search region based on the key feature sequences includes: extracting key feature sequences between the video front and back frames, and performing time sequence tracking on the key feature sequences by using the bidirectional long short-term memory network to obtain a time-domain-based Token sequence, wherein the time-domain-based Token sequence is used to determine a motion vector search direction; predicting a position of the target search region in a next video frame based on the motion vector search direction and the key feature sequences; and outputting a coordinate parameter of the target search region in the next video frame based on the foreground region segmentation coordinate and the predicted position of the target search region in the next video frame.

[0011] Optionally, the step of extracting the key feature sequence between consecutive video frames includes: extracting local features from the previous video frame to obtain a first local feature set, and extracting local features from the next video frame to obtain a second local feature set, wherein the previous video frame and the next video frame are adjacent video frames; comparing the feature elements in the first local feature set and the second local feature set to determine the positions of the same feature elements in the previous video frame and the next video frame, respectively, to form a local feature sequence; and using an attention score mechanism to extract key features from the local feature sequence to generate the key feature sequence between consecutive video frames.

[0012] Optionally, the step of performing an encoder inter-frame macroblock motion vector analysis strategy within the target search area to determine the macroblock motion vectors within the target search area includes: using a preset random sampling consistency algorithm to remove non-salient feature points within the target search area; and performing an encoder inter-frame macroblock motion vector analysis strategy within the target search area of ​​the removed non-salient feature points to determine the macroblock motion vectors within the target search area.

[0013] According to another aspect of the present invention, a neural network-based inter-frame macroblock motion vector determination device is also provided, comprising: a video frame segmentation unit, configured to perform semantic segmentation on a video frame using an improved U2net neural network to determine a foreground region and a background region in the video frame; a search region determination unit, configured to analyze key feature sequences of consecutive frames of the video using a deep local feature search network based on reference objects in the foreground or background region, and determine a target search region based on the key feature sequences, wherein the deep local feature search network has incorporated a time-series neural network based on a bidirectional long short-term memory network, and the key feature sequences include: feature points of multiple key objects and preliminary motion vectors, wherein the key objects refer to objects whose similarity between consecutive frames of the video is greater than a preset similarity threshold; and a motion vector prediction unit, configured to execute an encoder inter-frame macroblock motion vector analysis strategy within the target search region to determine macroblock motion vectors within the target search region.

[0014] Optionally, the video frame segmentation unit includes: a feature extraction module, used to extract features from the video frame using a U-structured Encoder layer to obtain a feature map set; a first convolution module, used to perform dilated convolution operations at multiple scales on each feature map in the feature map set using an dilated spatial pyramid pooling structure (ASPP), and to perform upsampling operations to obtain a feature map set of the same size; a convolution dimensionality reduction module, used to perform convolution dimensionality reduction operations on the channel feature maps in the feature map set of the same size to obtain a 2D vector matrix; a normalization module, used to perform channel fusion and normalization processing on the elements in the 2D vector matrix, and output a probability value, wherein the probability value is used to indicate whether there is a foreground object in the video frame; and a video frame segmentation module, used to determine the region containing the foreground object as the foreground region and the other regions besides the foreground region as the background region when the probability value indicates that there is a foreground object in the video frame.

[0015] Optionally, the convolutional dimensionality reduction module includes: a nonlinear transformation submodule, used to perform nonlinear transformation processing on each channel feature map using a one-dimensional convolutional layer to obtain a converged 8-dimensional vector; and a convolutional dimensionality reduction submodule, used to perform convolutional dimensionality reduction operation on the converged 8-dimensional vector based on a two-layer convolutional loss function to obtain a 2-dimensional vector matrix, wherein the two-layer convolutional loss function integrates the loss function of the U-structure Encoder layer, the image texture feature loss function, and the overall image feature loss function.

[0016] Optionally, the inter-frame macroblock motion vector determination device based on neural networks further includes: an image coordinate origin determination unit, used to determine the image coordinate origin of the foreground region containing the foreground object after performing channel fusion and normalization processing on the elements in the 2D vector matrix and outputting probability values; and a foreground region segmentation coordinate output unit, used to construct a coordinate system with the image coordinate origin as the base point and output the foreground region segmentation coordinates.

[0017] Optionally, the search region determination unit includes: a key feature sequence extraction module, used to extract key feature sequences between consecutive frames of the video, and use the bidirectional long short-term memory network to perform temporal tracking on the key feature sequences to obtain a temporal-domain-based token sequence, wherein the temporal-domain-based token sequence is used to determine the motion vector search direction; a region position prediction module, used to predict the position of the target search region in the next video frame based on the motion vector search direction and the key feature sequence; and a search region coordinate parameter output module, used to output the coordinate parameters of the target search region in the next video frame based on the foreground region segmentation coordinates and the predicted position of the target search region in the next video frame.

[0018] Optionally, the key feature sequence extraction module includes: a local feature extraction submodule, used to extract local features from the previous video frame to obtain a first local feature set, and extract local features from the next video frame to obtain a second local feature set, wherein the previous video frame and the next video frame are adjacent video frames; a feature comparison submodule, used to compare feature elements in the first local feature set and the second local feature set, determine the positions of the same feature elements in the previous video frame and the next video frame respectively, and form a local feature sequence; and an attention extraction submodule, used to extract key features from the local feature sequence using an attention score mechanism to generate a key feature sequence between the preceding and following video frames.

[0019] Optionally, the motion vector prediction unit includes: an extraction module, used to remove non-salient feature points within the target search area using a preset random sampling consistency algorithm; and a motion vector determination module, used to perform an encoder inter-frame macroblock motion vector analysis strategy within the target search area of ​​the removed non-salient feature points to determine the macroblock motion vectors within the target search area.

[0020] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute any of the above-described neural network-based inter-frame macroblock motion vector determination methods.

[0021] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the neural network-based inter-frame macroblock motion vector determination method described above.

[0022] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the inter-frame macroblock motion vector determination method based on neural networks as described above.

[0023] In this disclosure, an improved U2net neural network is used to perform semantic segmentation on video frames to determine the foreground and background regions in the video frames. Based on reference objects in the foreground or background regions, a deep local feature search network is used to analyze the key feature sequences of consecutive video frames, and the target search region is determined based on the key feature sequences. The deep local feature search network has integrated a time-series neural network based on a bidirectional long short-term memory network. The key feature sequences include feature points of multiple key objects and preliminary motion vectors. Key objects refer to objects whose similarity between consecutive video frames is greater than a preset similarity threshold. An encoder inter-frame macroblock motion vector analysis strategy is executed within the target search region to determine the macroblock motion vectors within the target search region.

[0024] Based on the above disclosure, video frames can be segmented using a U2net neural network to obtain the region where the reference object is located. Then, a local feature search neural network (a time-series neural network that integrates a bidirectional long short-term memory network) can be used to fuse the similarity between consecutive frames, greatly improving the range of local feature search and effectively obtaining the similarity region of the image. Finally, the macroblock motion vector is determined in this region. The selected search region is more accurate, which can improve the prediction accuracy of the motion vector. This solves the technical problem in related technologies where using local features for MV prediction during video frame encoding can easily lead to inappropriate selection of the search range, affecting the prediction accuracy. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0026] Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a neural network-based inter-frame macroblock motion vector determination method is shown.

[0027] Figure 2 This is a flowchart of an optional inter-frame macroblock motion vector determination method based on a neural network according to an embodiment of the present invention;

[0028] Figure 3 This is an improved version U according to an embodiment of the present invention. 2 A schematic diagram of the net dual-head output neural network model;

[0029] Figure 4 This is a U according to an embodiment of the present invention. 2 A schematic diagram illustrating Network 2 (the main improvement part) in the net dual-head output neural network model;

[0030] Figure 5This is a schematic diagram illustrating how semantic segmentation determines the presence of a foreground image and obtains segmentation coordinates according to an embodiment of the present invention.

[0031] Figure 6 This is a schematic diagram illustrating the search range for calculating the MV by capturing key points of the foreground and background according to an embodiment of the present invention;

[0032] Figure 7 This is a schematic diagram of an improved DELF neural network model fused with LSTM according to an embodiment of the present invention;

[0033] Figure 8 This is a flowchart illustrating the timing of two neural networks calling a possible HEVC inter-frame prediction MV calculation process according to an embodiment of the present invention.

[0034] Figure 9 This is a schematic diagram of an optional inter-frame macroblock motion vector determination device based on a neural network according to an embodiment of the present invention;

[0035] Figure 10 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0036] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0038] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0039] H265 / HEVC: an abbreviation for High Efficiency Video Coding.

[0040] MV: an abbreviation for Motion Vector. In video coding, MV represents the positional offset between a block in the current frame and the same or most similar block in the reference frame, and is used to describe the motion of an object.

[0041] CU: Abbreviation for Coding Unit, is the basic processing unit used for video coding in HEVC. A CU can be further subdivided into multiple PUs (Prediction Units) and TUs (Transform Units). A PU is the unit used for prediction in HEVC; it can spatially partition a CU for more efficient inter-frame and intra-frame prediction. A TU is the unit used for transformation and quantization processes in HEVC; it can further subdivide a PU.

[0042] AMVP: an abbreviation for Advanced Motion Vector Prediction, used to reduce the motion vector coding overhead of inter-frame prediction.

[0043] DELF: an abbreviation for Deep Local Feature, is a deep learning model used for image retrieval that can extract and match local features of an image.

[0044] LSTM: an abbreviation for Long Short-Term Memory, is a type of recurrent neural network that can learn long-term dependencies and is suitable for processing sequential data.

[0045] RANSAC: an abbreviation for Random Sample Consensus, is an algorithm used to estimate parametric models, especially when outliers exist in the data, it can find the optimal model parameters.

[0046] SIFT is an abbreviation for Scale-Invariant Feature Transform.

[0047] Sobel Operator: An edge detection operator used in image processing and computer vision to detect significant changes in pixel intensity in an image.

[0048] Bi-LSTM: an abbreviation for Bidirectional Long Short-Term Memory, is a bidirectional long short-term memory network that can process sequential data in two directions, thereby capturing more comprehensive contextual information.

[0049] ASPP: An abbreviation for Atrous Spatial Pyramid Pooling, is a deep learning architecture used for image classification and segmentation that can capture image features at different scales.

[0050] CUDA: An abbreviation for Compute Unified Device Architecture.

[0051] NPU: an abbreviation for Neural Processing Unit, is a chip specifically designed for processing neural network calculations.

[0052] CTU: an abbreviation for Coding Tree Unit, which is the initial unit used for image segmentation in HEVC.

[0053] It should be noted that the inter-frame macroblock motion vector determination method and apparatus based on neural networks in this disclosure can be used in the field of video processing technology for the prediction of inter-frame macroblock motion vectors of encoders based on neural networks, and can also be used in any field other than video processing technology for the prediction of inter-frame macroblock motion vectors of encoders based on neural networks. This disclosure does not limit the application field of the inter-frame macroblock motion vector determination method and apparatus based on neural networks.

[0054] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) collected in this public disclosure are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0055] It should be noted that in this disclosure, customer information is collected and analyzed, and users are provided with corresponding operation entry points to choose whether to agree to or reject the automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0056] The following embodiments of the present invention can be applied to various systems / applications / devices for determining inter-frame macroblock motion vectors based on neural networks. The present invention can be applied to various video encoding scenarios, especially to high-efficiency video encoding hardware such as H.265 or H.264 encoders. Specific applications include high-definition video surveillance systems, video encoding chip design, online video streaming services, and, for example, the development of next-generation video encoding hardware. It integrates neural network capabilities, improving encoding efficiency while reducing chip power consumption and size, meeting the needs of portable and battery-powered devices.

[0057] It's important to note that inter-frame prediction in HEVC essentially involves the current frame's PU (Programmable Component) searching for the most similar macroblock in the reference frame. Since reference images are typically quite large, a global search can be very time-consuming. Therefore, a method should be used to determine a starting point in the reference image, and then the search should proceed around that starting point, thus reducing the search time. Currently, the mainstream HEVC methods employ two modes: AMVP and merge, which will be briefly explained below.

[0058] (1) Before starting the search, the AMVP mode first predicts an MV for the current PU. This predicted MV (denoted as MVP) can be directly obtained from the neighboring blocks in the spatial or temporal domain. Since there are multiple neighboring blocks, there will also be multiple MVPs. These MVPs form an MVP candidate list. The best one is selected as the actual MVP. The MVP point is used as the search starting point. The search is performed in the vicinity of the starting point according to a certain method (there are many search topologies, which will not be elaborated here). Finally, an optimal MV is obtained. This MV is the actual MV. The MV determines the position of the reference block. The reference block is subtracted from the current PU to obtain the residual, thus achieving the purpose of data compression.

[0059] (2) The merge mode is similar to AMVP. It obtains an MVP candidate list through adjacent blocks, selects the best MVP as the MV of the current block, and then directly determines the position of the reference block based on the MV. After determining the reference block, the residual can be calculated. The purpose of inter-frame prediction is to obtain an MV (motion vector) and then determine the position of the reference block in the reference image based on the MV. However, due to the similarity of neighboring blocks (for example, the current block and the neighboring blocks belong to the same object, and the distance and direction of the whole frame movement are of course the same when the video is playing), it is often not necessary to calculate the MV. The MV of the neighboring block is directly used as the MV of the current block. If the MVP and MV are the same, there is no MVD. Therefore, during encoding, it is only necessary to encode the index of the MV (MVP) in the candidate list. It is no longer necessary to encode the MVD. Decoding can construct the MVP candidate list in a similar way, and then obtain the MV based on the transmitted index.

[0060] This invention provides a method for locating the search region in macroblock mode selection during inter-frame prediction in an HEVC video encoder. Inter-frame prediction in an H.265 encoder is achieved by analyzing the PU unit with the smallest bitstream among candidate MVs in the spatial and temporal domains of surrounding macroblocks, and finally selecting the MV with the lowest rate-distortion cost. However, this method can only be used to search and predict in the region surrounding the macroblock. In this invention, by analyzing the similarity of consecutive frames, the corresponding PU blocks with relatively low rate-distortion costs are found, which can effectively reduce the bitrate. The image similarity adopts a two-model fusion method, including: the first is an algorithm based on smooth image similarity region analysis without foreground images, and the second is an algorithm with foreground images for similar region coordinates. By fusing these two methods, the PU search region selection of the image can be effectively improved. First, a local feature search neural network is used (this invention improves the DELP network and adds an LSTM-based time series neural network to fuse the similarity of consecutive frames, which greatly improves the range of local feature search). Then, the obtained local feature sequence is used to delete non-salient feature points using the RANSAC algorithm to effectively obtain the similarity region of the image. Finally, the PU prediction block is searched in this region to finally determine the structure of the CU in the CTU.

[0061] Secondly, objects with foreground are extracted using a U2net semantic segmentation neural network to obtain the region where key content is located. After separating the foreground and background content, similarity analysis is performed again on the local feature points in the key region using method one to obtain the direction of the search region, calculate the rate-distortion cost of the corresponding CU, and finally determine the MV direction parameters of the encoding.

[0062] This invention can optimize motion vector prediction strategies by intelligently analyzing the foreground and background in video frames, reducing unnecessary search calculations and thus improving video encoding efficiency. While ensuring video quality, it significantly reduces the bit rate of video encoding, thereby reducing network transmission costs and storage space requirements. It is especially suitable for large-scale video data processing and distribution.

[0063] In addition, this invention can improve the detail retention and visual clarity of key areas, such as moving people or objects, by prioritizing the encoding of these areas, thereby enhancing the visual effects and resolution of video content. In particular, the neural network's analysis and prediction of motion features can better adapt to the dynamic changes of video content, ensuring good encoding results even in fast-moving scenes.

[0064] The present invention will now be described in detail with reference to various embodiments.

[0065] Example 1

[0066] According to an embodiment of the present invention, an embodiment of a method for determining inter-frame macroblock motion vectors based on neural networks is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0067] The inter-frame macroblock motion vector determination method based on neural networks provided in Embodiment 1 of the present invention can be executed in a mobile terminal, computer terminal or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a neural network-based inter-frame macroblock motion vector determination method is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1 The processor 102 (which may include, but is not limited to, a microprocessor MCU (Microcontroller Unit) or a programmable gate array (FPGA)) is shown as 102a, 102b, ..., 102n. It also includes a memory 104 for storing data and a transmission device 106 for communication functions. In addition, it may include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0068] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of the present invention, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0069] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the inter-frame macroblock motion vector determination method based on neural networks in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned inter-frame macroblock motion vector determination method based on neural networks. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0070] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0071] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0072] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for determining inter-frame macroblock motion vectors based on neural networks is shown. Figure 2 This is a flowchart of an optional inter-frame macroblock motion vector determination method based on a neural network according to an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps:

[0073] Step S201: Use an improved U2net neural network to perform semantic segmentation on the video frame to determine the foreground and background regions in the video frame.

[0074] In this embodiment of the invention, step S201 describes the detailed process of performing semantic segmentation on video frames using an improved U2net neural network to distinguish between foreground and background regions. Specifically, the semantic segmentation process includes feature extraction, multi-scale analysis, feature fusion and dimensionality reduction, and the final probability value output. Each step is designed to more accurately identify key objects in the video, thereby optimizing subsequent video encoding and processing.

[0075] Optionally, step S201 includes: extracting features from the video frame using a U-structured Encoder layer to obtain a feature map set; performing dilated spatial pyramid pooling (ASPP) operations on each feature map in the feature map set using multiple scales of dilated convolution and upsampling to obtain a feature map set of the same size; performing convolution dimensionality reduction operations on the channel feature maps in the feature map set of the same size to obtain a 2D vector matrix; performing channel fusion and normalization on the elements in the 2D vector matrix to output a probability value, wherein the probability value is used to indicate whether there is a foreground object in the video frame; when the probability value indicates that there is a foreground object in the video frame, the region containing the foreground object is determined as the foreground region, and the other regions besides the foreground region are determined as the background region.

[0076] The video frames are first processed through the U-structure Encoder layer of the improved U2net neural network. This layer leverages the characteristics of deep convolutional neural networks to extract a set of multi-scale feature maps from the video frames. Each feature map represents different image features, including low-level texture and color information, as well as higher-level shape and structural information. These feature map sets form the basis for subsequent processing. It is worth noting that in the U-structure Encoder layer, the video frames undergo a feature extraction process from low to high and then from high to low, forming a set of feature maps containing multiple scales. This process enables the model to understand the image from multiple perspectives, not only capturing the local features of objects but also understanding the overall layout and position of objects in the image.

[0077] The extracted feature map set is then fed into an ASPP (Atrous Spatial Pyramid Pooling) structure for multi-scale feature fusion. ASPP captures details and background information in the image by performing dilated convolutions at different scales, effectively recognizing objects even when their size and position vary. Specifically, each feature map undergoes dilated convolution operations with different dilation rates. These operations expand the field of view of the convolution kernel without increasing the number of parameters or computational cost. Subsequently, all processed feature maps are upsampled to the same size, forming a new, uniformly sized feature map set for subsequent feature fusion and dimensionality reduction. The ASPP structure effectively addresses the uncertainty of object size and position in video frames, ensuring the model can accurately identify and locate foreground objects in various scenes. By performing dilated convolutions at different dilation rates, the network can simultaneously capture object details and background environmental information, which is crucial for understanding the specific state and position of objects in the scene.

[0078] To further reduce the dimensionality of the feature map set while preserving key information, this embodiment of the invention employs convolutional dimensionality reduction. First, convolutional dimensionality reduction is performed on the feature map of each channel, transforming the multi-dimensional feature map into a two-dimensional vector matrix through a one-dimensional convolutional layer. Each vector element carries rich information about the features at that location. To ensure the accuracy and robustness of this transformation process, multi-level loss functions are introduced, including the U-structured Encoder layer loss, image texture feature loss, and overall image feature loss. These loss functions collectively guide the model training and optimize feature extraction performance. Finally, channel fusion is performed on the generated two-dimensional vector matrix, merging the vector elements from different channels and normalizing them using a function such as Softmax. This outputs a probability value indicating the presence of foreground objects in the video frame. This probability value directly reflects whether a specific location in the video frame contains foreground object information, thus enabling intelligent foreground / background identification.

[0079] The step of performing convolutional dimensionality reduction on channel feature maps in a feature map set of the same size to obtain a 2D vector matrix includes: performing nonlinear transformation on each channel feature map using a one-dimensional convolutional layer to obtain a converged 8D vector; and performing convolutional dimensionality reduction on the converged 8D vector based on a two-layer convolutional loss function to obtain a 2D vector matrix. The two-layer convolutional loss function integrates the loss function of the U-structure Encoder layer, the image texture feature loss function, and the overall image feature loss function.

[0080] In the nonlinear transformation process, each channel feature map needs to be compressed and nonlinearly mapped using a one-dimensional convolutional layer. The one-dimensional convolutional layer operates along the channel dimension, compressing the feature map from multi-dimensional to one-dimensional, generating a vector containing important information. This process can be viewed as a form of feature selection, integrating multi-channel depth information into a more compact feature vector, thereby accelerating subsequent processing steps and reducing computational resource consumption. Each feature map is converted into an 8-dimensional vector, which constitutes a refined representation of the feature map set. To further fuse and reduce the dimensionality of these 8-dimensional vectors, this embodiment uses a loss function based on two convolutional layers. This loss function comprehensively considers the loss of the U-structure encoder layer, image texture feature loss, and overall image feature loss. Through this multi-objective loss function, the model can learn a more comprehensive and in-depth feature representation during training. Based on this loss function, the 8-dimensional vector is further convolved for dimensionality reduction, ultimately obtaining a 2-dimensional vector matrix. Each element of this matrix represents the comprehensive feature representation of the corresponding region in the video frame.

[0081] In one optional embodiment, after performing channel fusion and normalization on the elements in the 2D vector matrix and outputting probability values, the method further includes: determining the image coordinate origin of the foreground region containing the foreground object; constructing a coordinate system based on the image coordinate origin and outputting the foreground region segmentation coordinates.

[0082] Based on the coordinate system, the segmentation information output by the improved U2net neural network is used to accurately output the segmentation coordinates of the foreground region. This coordinate information is the key to subsequent processing (such as local feature point extraction, motion analysis, etc.), ensuring that the algorithm processes the foreground region efficiently and accurately.

[0083] Step S202: Based on reference objects in the foreground or background region, a deep local feature search network is used to analyze the key feature sequences of the preceding and following frames of the video, and the target search region is determined based on the key feature sequences. The deep local feature search network has been integrated with a time-series neural network based on a bidirectional long short-term memory network. The key feature sequences include: feature points of multiple key objects and preliminary motion vectors. Key objects refer to objects whose similarity between preceding and following frames of the video is greater than a preset similarity threshold.

[0084] Among them, the deep local feature search network can adopt an architecture based on DELF (Deep Local Feature), which, by fusing a bidirectional long short-term memory network (Bi-LSTM), can capture the temporal dependence of object motion in video sequences, as well as the local feature correlations between consecutive frames. Bi-LSTM propagates information forward and backward in the feature sequence, so that the feature representation at each time point not only depends on the current frame, but also takes into account the information of historical frames and the next frame, thus improving the accuracy of object tracking and prediction.

[0085] The key feature sequence includes not only feature points of multiple key objects but also preliminary motion vector information. Key objects refer to those objects with high similarity between consecutive frames of the video. Based on a deep local feature search network, by analyzing the key feature points of consecutive frames, the preliminary motion trend of objects can be predicted. These feature points are usually salient points or corners on the object's surface, and their changes between frames reflect the object's movement. The preliminary motion vector is the motion vector initially determined by the feature points of the key objects, calculated based on the displacement of the feature points. It is used to guide subsequent accurate motion vector prediction. In subsequent motion vector prediction, the preliminary motion vector can be used directly, or the encoder's inter-frame macroblock motion vector analysis strategy can be executed within the target search area to accurately determine the macroblock motion vectors within the target search area.

[0086] Optionally, the step of using a deep local feature search network to analyze the key feature sequences of consecutive video frames and determining the target search region based on the key feature sequences includes: extracting the key feature sequences between consecutive video frames and using a bidirectional long short-term memory network to perform temporal tracking of the key feature sequences to obtain a temporal-domain-based token sequence, wherein the temporal-domain-based token sequence is used to determine the motion vector search direction; predicting the position of the target search region in the next video frame based on the motion vector search direction and the key feature sequences; and outputting the coordinate parameters of the target search region in the next video frame based on the foreground region segmentation coordinates and the predicted position of the target search region in the next video frame.

[0087] The step of extracting key feature sequences between consecutive video frames includes: extracting local features from the previous video frame to obtain a first local feature set, and extracting local features from the subsequent video frame to obtain a second local feature set, wherein the previous and subsequent video frames are adjacent video frames; comparing the feature elements in the first and second local feature sets to determine the positions of the same feature elements in the previous and subsequent video frames, respectively, to form a local feature sequence; and using an attention score mechanism to extract key features from the local feature sequence to generate a key feature sequence between consecutive video frames.

[0088] First, local features are extracted from adjacent preceding and following video frames to generate a first local feature set and a second local feature set. This process utilizes the feature extraction capabilities of deep learning models (such as the DELF network) to identify key local features in the video frames, such as textures, edges, and object contours, ensuring the representativeness and relevance of the features. Next, the corresponding feature elements in the first and second local feature sets are compared to determine the positional changes of these features in the preceding and following video frames, thus forming a local feature sequence. This local feature sequence is the foundation of motion information between video frames; it contains the motion trajectory of video content between consecutive temporally connected frames and is crucial for predicting motion vectors (MV) and determining the search region. Then, using the Attention Score mechanism, this embodiment further filters key features from the local feature sequence to generate a key feature sequence between preceding and following video frames. The Attention Score mechanism identifies which features in the sequence are more important for determining motion vectors and the direction of the search region, and through weighted assignment, ensures that these key features dominate subsequent analysis.

[0089] It should be noted that this embodiment employs a bidirectional long short-term memory (Bi-LSTM) network to perform temporal tracking of key feature sequences, obtaining a time-domain-based token sequence. The Bi-LSTM network can simultaneously perform information transfer and feature learning in both directions of the video frame sequence. In this way, the token sequence contains dynamic change information of key features between consecutive frames, providing strong support for determining the search direction of motion vectors. Based on the motion vector search direction and the key feature sequence, this embodiment predicts the position of the target search region in the next video frame. During the prediction process, the motion trajectory information encoded in the token sequence is utilized, intelligently determining the approximate range of the search region and avoiding the high complexity and low efficiency of global search in traditional methods. Finally, combined with the foreground region segmentation coordinates, the coordinate parameters of the target search region in the next video frame are output, guiding the encoder's search strategy during inter-frame prediction. This plays a decisive role in improving encoding efficiency and video quality.

[0090] Step S203: Execute the encoder inter-frame macroblock motion vector analysis strategy within the target search area to determine the macroblock motion vectors within the target search area.

[0091] Optionally, the step of performing an encoder inter-frame macroblock motion vector analysis strategy within the target search area to determine the macroblock motion vectors within the target search area includes: using a preset random sampling consistency algorithm to remove non-salient feature points within the target search area; and performing an encoder inter-frame macroblock motion vector analysis strategy within the target search area of ​​the removed non-salient feature points to determine the macroblock motion vectors within the target search area.

[0092] This embodiment employs a pre-defined Random Sample Consensus (RANSAC) algorithm to eliminate non-salient feature points within the target search region (i.e., the foreground region). RANSAC is a robust statistical method used to estimate parametric models, even when the dataset contains a large amount of noise (outliers). In video coding scenarios, RANSAC can be used to select truly salient feature points representing object motion from a series of candidate feature points, excluding those non-salient points caused by environmental changes or noise, thereby improving the accuracy of motion vector prediction. During motion estimation and prediction in video coding, video sequences may contain a large number of unstable feature points or noise. These interfering factors can mislead motion vector prediction, leading to unnecessary coding redundancy and reduced prediction accuracy. By eliminating such non-salient points using the RANSAC algorithm and retaining only feature points that truly reflect the object's trajectory, the focus and effectiveness of the motion vector analysis strategy can be ensured, avoiding the encoding of irrelevant or erroneous information, and greatly improving the bitstream control capability and transmission efficiency of video coding.

[0093] Through the above steps, an improved U2net neural network can be used to perform semantic segmentation on video frames, determining the foreground and background regions in the video frames. Based on reference objects in the foreground or background regions, a deep local feature search network is used to analyze the key feature sequences of consecutive video frames, and the target search region is determined based on the key feature sequences. The deep local feature search network has integrated a time-series neural network based on a bidirectional long short-term memory network. The key feature sequences include feature points of multiple key objects and preliminary motion vectors. Key objects refer to objects whose similarity between consecutive video frames is greater than a preset similarity threshold. Within the target search region, an encoder inter-frame macroblock motion vector analysis strategy is executed to determine the macroblock motion vectors within the target search region. In this embodiment, a U2net neural network can be used to segment video frames and obtain the region where the reference object is located. Then, a local feature search neural network (a time-series neural network that integrates a bidirectional long short-term memory network) is used to fuse the similarity between consecutive frames, thereby increasing the range of local feature search and effectively obtaining the similarity region of the image. Finally, the macroblock motion vector is determined in this region. The selected search region is more accurate, which can improve the prediction accuracy of the motion vector. This solves the technical problem in related technologies where using local features for MV prediction during video frame encoding can easily lead to improper selection of the search range and affect the prediction accuracy.

[0094] The following describes in detail another optional implementation method.

[0095] This invention provides a neural network-based inter-frame macroblock motion vector prediction method for H.265 encoders, achieving MV prediction. It involves the improved design of two neural networks, the first being based on U... 2 The first type is an improved semantic segmentation network, and the second type is an improved neural network based on Bi-LSTM and DELF. The following is a schematic description of each of these improved neural networks.

[0096] Figure 3 This is an improved version U according to an embodiment of the present invention. 2 A schematic diagram of the net dual-head output neural network model, as shown below. Figure 3 As shown, Network 1 on the right represents the original neural network, while Network 2 on the left represents the improved part (green boxes indicate the presence of foreground objects, and purple boxes indicate the absence of foreground objects).

[0097] Figure 4 This is a U according to an embodiment of the present invention. 2 A schematic diagram illustrating Network 2 (the main improvement part) in the net dual-head output neural network model.

[0098] U based on semantic segmentation 2The net neural network acquires foreground and background regions as data sources for the next step of local feature extraction. The designed network model can effectively segment the foreground and background regions. Based on the improved U... 2 The net network performs semantic segmentation and can effectively distinguish whether there are foreground objects. If there are no foreground objects, the net network will give a softmax normalized 2D vector indicating whether the foreground exists.

[0099] like Figure 4 As shown, the red area network 2 represents the improved version. The algorithm is explained below: The image of the boat in the lake uses the U-structure encoder to split the feature layer image. Then, the ASPP (spatial pyramid pooling layer) operator is used to convert the 5-layer feature map into a 2-bit fixed vector. First, the 5-layer feature map is subjected to dilated convolution operations at different scales, then upsampled into 5 sets of feature maps of the same size. Then, 1x1 convolution is used to reduce the dimensionality to 2 channels. Then, the full-size feature map is used as the convolution kernel to perform convolution, the purpose of which is to convert the 2-dimensional feature map into a 1-dimensional vector. After finally converting into 2 feature vectors, the cencatenate operator is used to fuse the 2-bit vectors again, and then softmax is used to normalize them into probability values, giving the probability of whether a foreground image exists in the two categories (foreground image exists, foreground image does not exist).

[0100] Figure 5 This is a schematic diagram illustrating how semantic segmentation determines the presence of a foreground image and obtains segmentation coordinates according to an embodiment of the present invention. Figure 5 As shown, the coordinate parameters of the cruise ship in the foreground area of ​​the image are determined.

[0101] This algorithm adds the loss of the encoder layer and the 8-bit 1D vector layer after ASPP downsampling to the U2net loss function, as shown in the formula below:

[0102]

[0103] VGG is the loss of the feature extraction layer, SSIM is the texture feature loss, and MSE is the overall feature loss (PSNR can also be used instead). The loss function in this design is a Level 2 layer, which includes two types of losses: the first is the classification loss function, and the second is the ASPP supervision loss function. The two parts of the loss account for 70% and 30% of the weight, respectively, and finally form the loss part for classifying invalid images.

[0104] Index region selection based on key feature points is achieved using a DELF neural network based on LSTM in a multi-frame mode. Figure 6The method shows that the location of the feature points of different objects in the foreground and background changes significantly. For unimportant background objects, the search area is obtained through similar features to determine the MVP, and the calculated residual will be minimized. This effectively reduces the bitrate of the background content. For important parts of the foreground, the effect of camera tracking is that they are basically always kept in the center of the shot. Therefore, the search area of ​​the reference frame is much closer to the background reference frame area. This effectively solves the problem of the range of the search area for selecting reference frames for the foreground and background.

[0105] Figure 6 This is a schematic diagram illustrating the search range for calculating the MV by capturing key points in the foreground and background according to an embodiment of the present invention, as shown below. Figure 6 As shown, by using the coordinates of the black cruise ship in the foreground area (as a reference object), key points of the white cruise ship in the background area can be captured (such as...). Figure 6 (The parts marked with red lines in numbers 3 and 4 below).

[0106] Figure 7 This is a schematic diagram of an improved DELF neural network model incorporating LSTM according to an embodiment of the present invention, as shown below. Figure 7 As shown, it integrates DELF neural network and Bi-LSTM neural network. It extracts local features through DELF neural network, then uses attention score to extract salient features of image to generate key feature sequence. The extracted key feature sequence is input into the input layer of Bi-LSTM neural network. The input layer puts the sequence features into the front layer and the back layer respectively. Finally, the output layer outputs information about the search area.

[0107] This network can effectively extract salient feature points between consecutive frames for comparison, and track highly similar feature sequences (Tokens) using LSTM sequences to predict the location of the next search area.

[0108] DELF employs an attention score-based approach. Instead of directly using densely extracted features for image retrieval, it designs an attention mechanism to effectively select local features to form subsets, avoiding the computational burden caused by a large amount of dense extraction. Therefore, key point selection is crucial for both the accuracy and computational efficiency of the retrieval system.

[0109] The Bi-LSTM front layer processes the P-frame portion of the video sequence, while the back layer LSTM sequence processes the B-frame portion of the video sequence, thus covering most scenes with I, B, and P frames.

[0110]

[0111] In the formula, α is the attention score function, and the following AF function is used for feature attention discretization:

[0112]

[0113] The final loss function of DELF uses cross-entropy loss:

[0114]

[0115] The attention function u (i.e., θ) is trained through backpropagation, and the gradient is calculated using the following derivative equation:

[0116]

[0117] In the proposed framework, the parameters of the W function and the attention model parameters are implicitly learned through image-level labels. This learning process is somewhat challenging; the feature representation and attention score function can be jointly trained via backpropagation, potentially resulting in a weak model with extremely poor generalization ability. Therefore, a two-step training strategy can be adopted. First, hyperparameters for feature extraction are learned using conventional cross-entropy. Then, multiple shuffled batches are used to form new epochs, and repeated training and fine-tuning yield the basic hyperparameters. Finally, the parameters θ of the attention score network are trained based on these parameters, making the training process relatively easier.

[0118] Figure 8 This is a flowchart illustrating the timing of two neural networks calling a possible HEVC inter-frame prediction MV calculation process according to an embodiment of the present invention, as shown below. Figure 8 As shown, two types of neural networks are involved: one is the improved U2net neural network for semantic segmentation, and the other is the improved DELP neural network based on LSTM. The timing flowchart also integrates the frame classification, P-frame or B-frame prediction process and MV prediction process performed in advance by other encoders (such as H.264 encoder). The traditional AMVP or Merge algorithm is used for MV prediction. Then, the improved U2net network is used to extract the foreground and background regions, calculate the similarity of each object in the region, and then the improved DELF network is used to search for the MV region, return the coordinates of the search region, and finally calculate the HEVC syntax elements.

[0119] Through the above implementation method, objects with foreground can be extracted using a U2net semantic segmentation neural network to obtain the region where key content is located. After separating the foreground and background content, similarity analysis is performed again on local feature points within the key region using method one to obtain the direction of the search region, calculate the rate-distortion cost of the corresponding CU, and finally determine the MV direction parameters of the encoding. A local feature search neural network is then used (this invention improves the DELP network design and adds an LSTM-based time series neural network to fuse the similarity between consecutive frames, increasing the range of local feature search). The obtained local feature sequence is then processed using the RANSAC algorithm to delete non-salient feature points, effectively obtaining the similarity region of the image. Finally, the PU prediction block is searched in this region to determine the structure of the CU in the CTU, thereby providing inter-frame search region selection for the HEVC encoder and improving search efficiency.

[0120] The following is a detailed description with reference to another embodiment.

[0121] Example 2

[0122] The inter-frame macroblock motion vector determination device based on neural networks provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in the above embodiment one. The specific implementation method and beneficial effects can be referred to the foregoing method embodiment, and will not be repeated here.

[0123] Figure 9 This is a schematic diagram of an optional inter-frame macroblock motion vector determination device based on a neural network according to an embodiment of the present invention, such as... Figure 9 As shown, the inter-frame macroblock motion vector determination device based on neural networks may include: a video frame segmentation unit 91, a search region determination unit 92, and a motion vector prediction unit 93.

[0124] Among them, the video frame segmentation unit 91 is used to perform semantic segmentation on video frames using an improved U2net neural network to determine the foreground and background regions in the video frames.

[0125] The search region determination unit 92 is used to analyze the key feature sequences of previous and subsequent video frames based on reference objects in the foreground or background regions using a deep local feature search network, and to determine the target search region based on the key feature sequences. The deep local feature search network has integrated a time series neural network based on a bidirectional long short-term memory network. The key feature sequences include feature points of multiple key objects and preliminary motion vectors. Key objects refer to objects whose similarity between previous and subsequent video frames is greater than a preset similarity threshold.

[0126] The motion vector prediction unit 93 is used to execute the encoder inter-frame macroblock motion vector analysis strategy within the target search area to determine the macroblock motion vectors within the target search area.

[0127] The aforementioned inter-frame macroblock motion vector determination device based on neural networks can use the video frame segmentation unit 91 to perform semantic segmentation on video frames using an improved U2net neural network to determine the foreground and background regions in the video frames. The search region determination unit 92 uses a deep local feature search network to analyze the key feature sequences of the preceding and following frames of the video based on reference objects in the foreground or background regions, and determines the target search region based on the key feature sequences. The deep local feature search network has integrated a time-series neural network based on a bidirectional long short-term memory network. The key feature sequences include feature points of multiple key objects and preliminary motion vectors. Key objects refer to objects whose similarity between preceding and following frames of the video is greater than a preset similarity threshold. The motion vector prediction unit 93 executes the encoder's inter-frame macroblock motion vector analysis strategy within the target search region to determine the macroblock motion vectors within the target search region. In this embodiment, a U2net neural network can be used to segment video frames and obtain the region where the reference object is located. Then, a local feature search neural network (a time-series neural network that integrates a bidirectional long short-term memory network) is used to fuse the similarity between consecutive frames, which greatly improves the range of local feature search and effectively obtains the similarity region of the image. Finally, the macroblock motion vector is determined in this region. The selected search region is more accurate, which can improve the prediction accuracy of the motion vector. This solves the technical problem in related technologies where using local features for MV prediction during video frame encoding can easily lead to improper selection of the search range and affect the prediction accuracy.

[0128] Optionally, the video frame segmentation unit includes: a feature extraction module, used to extract features from the video frame using a U-structured Encoder layer to obtain a set of feature maps; a first convolution module, used to perform dilated convolution operations at multiple scales on each feature map in the feature map set using an dilated spatial pyramid pooling structure (ASPP), and to perform upsampling operations to obtain a set of feature maps of the same size; a convolution dimensionality reduction module, used to perform convolution dimensionality reduction operations on the channel feature maps in the set of feature maps of the same size to obtain a 2D vector matrix; a normalization module, used to perform channel fusion and normalization processing on the elements in the 2D vector matrix, and output probability values, wherein the probability values ​​are used to indicate whether there are foreground objects in the video frame; and a video frame segmentation module, used to determine the region containing the foreground object as the foreground region and the other regions besides the foreground region as the background region when the probability value indicates that there are foreground objects in the video frame.

[0129] Optionally, the convolutional dimensionality reduction module includes: a nonlinear transformation submodule, used to perform nonlinear transformation processing on the feature map of each channel using a one-dimensional convolutional layer to obtain a converged 8-dimensional vector; and a convolutional dimensionality reduction submodule, used to perform convolutional dimensionality reduction operation on the converged 8-dimensional vector based on a two-layer convolutional loss function to obtain a 2-dimensional vector matrix, wherein the two-layer convolutional loss function integrates the loss function of the U-structure Encoder layer, the image texture feature loss function, and the overall image feature loss function.

[0130] Optionally, the inter-frame macroblock motion vector determination device based on neural networks further includes: an image coordinate origin determination unit, used to determine the image coordinate origin of the foreground region containing the foreground object after performing channel fusion and normalization processing on the elements in the 2D vector matrix and outputting probability values; and a foreground region segmentation coordinate output unit, used to construct a coordinate system with the image coordinate origin as the base point and output the foreground region segmentation coordinates.

[0131] Optionally, the search region determination unit includes: a key feature sequence extraction module, used to extract key feature sequences between consecutive video frames, and to perform temporal tracking of the key feature sequences using a bidirectional long short-term memory network to obtain a temporal-domain-based token sequence, wherein the temporal-domain-based token sequence is used to determine the motion vector search direction; a region position prediction module, used to predict the position of the target search region in the next video frame based on the motion vector search direction and the key feature sequence; and a search region coordinate parameter output module, used to output the coordinate parameters of the target search region in the next video frame based on the foreground region segmentation coordinates and the predicted position of the target search region in the next video frame.

[0132] Optionally, the key feature sequence extraction module includes: a local feature extraction submodule, used to extract local features from the previous video frame to obtain a first local feature set, and extract local features from the next video frame to obtain a second local feature set, wherein the previous video frame and the next video frame are adjacent video frames; a feature comparison submodule, used to compare feature elements in the first local feature set and the second local feature set, determine the positions of the same feature elements in the previous video frame and the next video frame respectively, and form a local feature sequence; and an attention extraction submodule, used to extract key features from the local feature sequence using an attention score mechanism to generate a key feature sequence between consecutive video frames.

[0133] Optionally, the motion vector prediction unit includes: an extraction module for removing non-salient feature points within the target search area using a preset random sampling consistency algorithm; and a motion vector determination module for performing an encoder inter-frame macroblock motion vector analysis strategy within the target search area containing the removed non-salient feature points to determine the macroblock motion vectors within the target search area.

[0134] The aforementioned inter-frame macroblock motion vector determination device based on neural networks may further include a processor and a memory. The aforementioned video frame segmentation unit 91, search region determination unit 92, motion vector prediction unit 93, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0135] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters enables inter-frame macroblock motion vector prediction by a neural network-based encoder.

[0136] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0137] Example 3

[0138] Embodiments of this application may provide an electronic device. Figure 10 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Figure 10 As shown, the electronic device may include: one or more ( Figure 10 (Only one is shown) processor 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0139] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the inter-frame macroblock motion vector determination method and apparatus based on neural networks in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned inter-frame macroblock motion vector determination method based on neural networks. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0140] The processor can access information and applications stored in memory via a transmission device to perform the following steps: Semantic segmentation of video frames using an improved U2net neural network to determine foreground and background regions within the video frames; analysis of key feature sequences between consecutive video frames using a deep local feature search network based on reference objects in the foreground or background regions, and determination of the target search region based on these key feature sequences. The deep local feature search network incorporates a time-series neural network based on a bidirectional long short-term memory network. Key feature sequences include feature points of multiple key objects and preliminary motion vectors. Key objects refer to objects whose similarity between consecutive video frames exceeds a preset similarity threshold; and execution of an encoder inter-frame macroblock motion vector analysis strategy within the target search region to determine the macroblock motion vectors within the target search region.

[0141] Those skilled in the art will understand that Figure 10 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 10 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 10 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 10 The different configurations shown.

[0142] Those skilled in the art will understand that all or part of the steps in the various neural network-based inter-frame macroblock motion vector determination methods described above can be implemented by a program instructing the hardware of the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0143] Example 4

[0144] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the neural network-based inter-frame macroblock motion vector determination method provided in Embodiment 1.

[0145] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute any of the neural network-based inter-frame macroblock motion vector determination methods described in Embodiment 1 above.

[0146] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0147] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the inter-frame macroblock motion vector determination method based on neural networks described in various embodiments of this application.

[0148] This application also provides a computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the inter-frame macroblock motion vector determination method based on neural networks described in various embodiments of this application.

[0149] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0150] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0155] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for determining inter-frame macroblock motion vectors based on neural networks, characterized in that, include: Adopting an improved U 2 The net neural network performs semantic segmentation on video frames to determine the foreground and background regions in the video frames. This includes: extracting features from the video frames using a U-structured Encoder layer to obtain a set of feature maps; performing dilated spatial pyramid pooling (ASPP) operations on each feature map in the set of feature maps at multiple scales, followed by upsampling, to obtain a set of feature maps of the same size; and performing convolutional dimensionality reduction operations on the channel feature maps in the set of feature maps of the same size to obtain a 2D vector matrix, including: performing nonlinear transformation processing on each channel feature map using a one-dimensional convolutional layer to obtain a converged 8D vector; based on the two... A two-layer convolutional loss function is applied to the converged 8-dimensional vector to reduce its dimensionality, resulting in a 2-dimensional vector matrix. This two-layer convolutional loss function integrates the loss function of the U-structure Encoder layer, the image texture feature loss function, and the overall image feature loss function. Channel fusion and normalization are then applied to the elements of the 2-dimensional vector matrix, outputting probability values. These probability values ​​indicate the presence of foreground objects in the video frame. If the probability value indicates the presence of foreground objects in the video frame, the region containing the foreground object is defined as the foreground region, and the regions other than the foreground region are defined as the background region. Based on reference objects in the foreground or background region, a deep local feature search network is used to analyze the key feature sequences of consecutive frames in the video, and the target search region is determined based on the key feature sequences. The deep local feature search network has been integrated with a time series neural network based on a bidirectional long short-term memory network. The key feature sequences include: feature points of multiple key objects and preliminary motion vectors. The key objects refer to objects whose similarity between consecutive frames in the video is greater than a preset similarity threshold. An encoder inter-frame macroblock motion vector analysis strategy is executed within the target search area to determine the macroblock motion vectors within the target search area.

2. The method according to claim 1, characterized in that, After performing channel fusion and normalization on the elements of the 2D vector matrix and outputting probability values, the process also includes: Determine the image coordinate origin of the foreground region containing the foreground object; A coordinate system is constructed using the origin of the image coordinates as the base point, and the foreground region segmentation coordinates are output.

3. The method according to claim 1, characterized in that, The steps of analyzing key feature sequences of consecutive frames in a video using a deep local feature search network and determining the target search region based on the key feature sequences include: Key feature sequences between consecutive frames of the video are extracted, and the bidirectional long short-term memory network is used to perform temporal tracking on the key feature sequences to obtain a time-domain-based token sequence, wherein the time-domain-based token sequence is used to determine the motion vector search direction; Based on the motion vector search direction and the key feature sequence, predict the position of the target search region in the next video frame; Based on the foreground region segmentation coordinates and the predicted position of the target search region in the next video frame, the coordinate parameters of the target search region in the next video frame are output.

4. The method according to claim 3, characterized in that, The step of extracting key feature sequences between consecutive frames of the video includes: Local features are extracted from the previous video frame to obtain a first set of local features, and local features are extracted from the next video frame to obtain a second set of local features, wherein the previous video frame and the next video frame are adjacent video frames. By comparing the feature elements in the first local feature set and the second local feature set, the positions of the same feature elements in the previous video frame and the positions in the next video frame are determined, forming a local feature sequence; The attention score mechanism is used to extract key features from the local feature sequence to generate a key feature sequence between consecutive frames of the video.

5. The method according to claim 1, characterized in that, The steps of performing an encoder inter-frame macroblock motion vector analysis strategy within the target search area to determine the macroblock motion vectors within the target search area include: A preset random sampling consistency algorithm is used to remove non-significant feature points within the target search area; An encoder inter-frame macroblock motion vector analysis strategy is performed within the target search region of the removed non-significant feature points to determine the macroblock motion vectors within the target search region.

6. A device for determining inter-frame macroblock motion vectors based on neural networks, characterized in that, include: Video frame segmentation unit, used to employ improved U 2 The net neural network performs semantic segmentation on video frames to determine the foreground and background regions in the video frames. The video frame segmentation unit includes: a feature extraction module, used to extract features from the video frame using a U-structured Encoder layer to obtain a feature map set; a first convolution module, used to perform dilated convolution operations at multiple scales on each feature map in the feature map set using an dilated spatial pyramid pooling structure (ASPP), and perform upsampling operations to obtain a feature map set of the same size; a convolution dimensionality reduction module, used to perform convolution dimensionality reduction operations on the channel feature maps in the feature map set of the same size to obtain a 2D vector matrix; and a normalization module, used to perform channel fusion and normalization processing on the elements in the 2D vector matrix and output probability values, wherein the probability values ​​are used to indicate whether there are foreground objects in the video frame. The video frame segmentation module is used to determine the region containing the foreground object as the foreground region and the other regions besides the foreground region as the background region when the probability value indicates that there is a foreground object in the video frame; the convolutional dimensionality reduction module includes: a nonlinear transformation submodule, used to perform nonlinear transformation processing on each channel feature map using a one-dimensional convolutional layer to obtain a converged 8-dimensional vector; and a convolutional dimensionality reduction submodule, used to perform convolutional dimensionality reduction operation on the converged 8-dimensional vector based on a two-layer convolutional loss function to obtain a 2-dimensional vector matrix, wherein the two-layer convolutional loss function integrates the loss function of the U-structure Encoder layer, the image texture feature loss function, and the overall image feature loss function; The search region determination unit is used to analyze the key feature sequences of consecutive frames of a video using a deep local feature search network based on reference objects in the foreground or background region, and to determine the target search region based on the key feature sequences. The deep local feature search network has been integrated with a time-series neural network based on a bidirectional long short-term memory network. The key feature sequences include feature points of multiple key objects and preliminary motion vectors. The key objects refer to objects whose similarity between consecutive frames of a video is greater than a preset similarity threshold. The motion vector prediction unit is used to perform an encoder inter-frame macroblock motion vector analysis strategy within the target search area to determine the macroblock motion vectors within the target search area.

7. An electronic device, characterized in that, The method includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the inter-frame macroblock motion vector determination method based on any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the inter-frame macroblock motion vector determination method based on any one of claims 1 to 5.

Images