Video encoding method, video decoding method, device and storage medium

Abstract

The application provides a video coding method, a video decoding method, a device and a storage medium. The video coding method comprises the following steps: acquiring a video frame sequence; calculating an inter-frame optical flow change index of any video frame in the video frame sequence; the inter-frame optical flow change index represents a motion change degree between the video frame and a previous video frame; dividing the video frame sequence into a key frame set and a redundant frame set; performing image coding on each key frame in the key frame set to obtain an image code stream; performing semantic text recognition on each redundant frame in the redundant frame set through multiple recognition models, and generating a semantic text code stream according to the one-to-one correspondence between the multiple redundant frames and the multiple semantic text recognition results; and sending the image code stream and the semantic text code stream to a cloud server. The application can flexibly adapt to different downstream tasks by extracting information of different dimensions.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a video encoding method, a video decoding method, an apparatus, and a storage medium. Background Technology

[0002] In an integrated edge-cloud intelligent sensing system, edge devices continuously generate massive amounts of video data, such as security surveillance videos and real-time traffic scene video streams. This video data is typically acquired and transmitted at high resolution and high frame rates, placing enormous pressure on the transmission and computation capabilities of both the edge and cloud sides. Traditional video coding methods, such as H.264 and H.265, primarily focus on pixel-level compression within frames or segments, but lack effective extraction of semantic and temporal information from the video. Therefore, these methods often result in the transmission of a large number of redundant video frames, increasing communication costs and reducing the efficiency of subsequent analysis.

[0003] To address this issue, existing technologies attempt to introduce feature encoding methods. These methods extract video features by deploying visual models at the edge and then transmit them, thereby reducing video transmission costs. However, this feature encoding method has significant limitations: it relies on a uniform and fixed model structure and feature representation, making it difficult to flexibly adapt to different temporal video tasks, such as video event detection, behavior recognition, and anomaly detection. Summary of the Invention

[0004] In view of this, this application proposes a video coding method, a video decoding method, an apparatus, and a storage medium to solve the problem that existing feature coding methods rely on a unified and fixed model structure and feature representation, making it difficult to flexibly adapt to different video temporal tasks.

[0005] The first aspect of this application proposes a video coding method, including: Obtain the video frame sequence; For any video frame in the video frame sequence, calculate the inter-frame optical flow variation index of the video frame; the inter-frame optical flow variation index represents the degree of motion change between the video frame and the previous video frame. The video frame sequence is divided into a key frame set and a redundant frame set; the inter-frame optical flow change index of the key frames in the key frame set is greater than or equal to the optical flow change threshold, and the inter-frame optical flow change index of the redundant frames in the redundant frame set is less than the optical flow change threshold. Each keyframe in the keyframe set is image encoded to obtain an image bitstream; Semantic text recognition is performed on each redundant frame in the redundant frame set using multiple recognition models, and a semantic text code stream is generated by combining the semantic text recognition results of multiple redundant frames. The image stream and the semantic text stream are sent to the cloud server.

[0006] This application embodiment uses semantic text code streams generated from multiple semantic text recognition results to replace redundant frame images for transmission, which can effectively reduce communication overhead; by performing semantic text recognition on each redundant frame in the redundant frame set using multiple recognition models, semantic information of different dimensions can be extracted, which can flexibly adapt to different downstream tasks and also help improve the accuracy of subsequent analysis tasks.

[0007] In this embodiment of the application, calculating the inter-frame optical flow variation index of the video frame includes: For any pixel among multiple pixels in the video frame, calculate the optical flow vector of the pixel between the video frame and the previous video frame; the optical flow vector represents the direction and distance of the pixel's movement between two adjacent video frames. Calculate the Euclidean norm of the optical flow vector; the Euclidean norm represents the motion amplitude of the pixel between two adjacent video frames; The summation result is obtained by summing the multiple Euclidean norms that correspond one-to-one with the multiple pixels; Based on the summation result and the total area of ​​the video frame, the inter-frame optical flow variation index of the video frame is calculated.

[0008] In this embodiment of the application, the video frame sequence is divided into a keyframe set and a redundant frame set, including: For any video frame in the video frame sequence, determine the relationship between the inter-frame optical flow change index and the optical flow change threshold of the video frame; If the inter-frame optical flow change index of the video frame is greater than or equal to the optical flow change threshold, then the video frame is determined as a key frame. If the inter-frame optical flow variation index of the video frame is less than the optical flow variation threshold, then the video frame is determined to be a redundant frame. The keyframe set is established based on multiple keyframes, and the redundant frame set is established based on multiple redundant frames.

[0009] In this embodiment, the multiple recognition models include a scene recognition model, an object detection model, and a text information extraction model; semantic text recognition is performed on each redundant frame in the redundant frame set using the multiple recognition models, including: For any redundant frame, the scene label information of the redundant frame is extracted according to the scene recognition model; The object information of the redundant frame is extracted according to the object detection model; the object information includes object attribute information and object location information. The text information of the redundant frames is extracted according to the text information extraction model; Determine the timestamp information for semantic text recognition of the redundant frames; The scene label information, the object information, the text information, and the timestamp information are combined to obtain the semantic text recognition result of the redundant frame.

[0010] In this embodiment of the application, calculating the inter-frame optical flow variation index of the video frame includes:

[0011] in, Indicates the first Inter-frame optical flow variation index for each video frame; Indicates the first The total area of ​​the image in each video frame. Indicates the height of the image. Indicates the width of the image; Indicates the first The video frame and the first Pixels between video frames The optical flow vector; The Euclidean norm of the optical flow vector is denoted by .

[0012] A second aspect of this application provides a video decoding method, including: The system receives an image stream and a semantic text stream. The image stream is obtained by encoding each keyframe in a keyframe set. The keyframe set consists of video frames in a video frame sequence whose inter-frame optical flow change index is greater than or equal to an optical flow change threshold. The inter-frame optical flow change index represents the degree of motion change between the current video frame and the previous video frame. The semantic text stream is generated based on multiple semantic text recognition results corresponding to multiple redundant frames. Each semantic text recognition result is obtained by performing semantic text recognition on the corresponding redundant frames in the redundant frame set using multiple recognition models. The redundant frame set consists of video frames in a video frame sequence whose inter-frame optical flow change index is less than the optical flow change threshold. The image stream is decoded to obtain multiple keyframes, and the semantic text stream is decoded to obtain multiple semantic text recognition results; The reconstructed video sequence information is obtained by combining the multiple keyframes and multiple semantic text recognition results.

[0013] In this embodiment of the application, after obtaining the reconstructed video sequence information, the method further includes: The reconstructed video sequence information is input into the trained abnormal event recognition model, and the abnormal event recognition result is output.

[0014] An embodiment of the third aspect of this application provides a video encoding apparatus, comprising: The sequence acquisition module is used to acquire video frame sequences; The index calculation module is used to calculate the inter-frame optical flow change index for any video frame in the video frame sequence; the inter-frame optical flow change index represents the degree of motion change between the video frame and the previous video frame. The sequence segmentation module is used to divide the video frame sequence into a key frame set and a redundant frame set; the inter-frame optical flow change index of the key frames in the key frame set is greater than or equal to the optical flow change threshold, and the inter-frame optical flow change index of the redundant frames in the redundant frame set is less than the optical flow change threshold. The image bitstream generation module is used to perform image encoding on each keyframe in the keyframe set to obtain an image bitstream; The semantic text stream generation module is used to perform semantic text recognition on each redundant frame in the redundant frame set using multiple recognition models, and to generate a semantic text stream from the multiple semantic text recognition results corresponding to multiple redundant frames. The code stream sending module is used to send the image code stream and the semantic text code stream to the cloud server.

[0015] An embodiment of the fourth aspect of this application provides an electronic device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first or second aspect above.

[0016] An embodiment of the fifth aspect of this application provides a computer-readable storage medium storing computer instructions that are used to cause a computer to perform the methods described in the first or second aspect above.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a video encoding method provided in an embodiment of this application is shown; Figure 2 A flowchart illustrating a video decoding method provided in an embodiment of this application is shown; Figure 3 This invention provides a schematic diagram of the structure of a video encoding apparatus according to an embodiment of the present application. Figure 4 This illustration shows a schematic diagram of the structure of an electronic device according to an embodiment of this application; Figure 5 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0019] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0020] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0021] According to an embodiment of this application, a video encoding method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0022] Example 1: This embodiment provides a video encoding method, which is applied to a client. Figure 1 This is a flowchart of a video encoding method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the video frame sequence.

[0023] Specifically, video data of real-time traffic scenes can be collected using camera sensors and parsed into video frame sequences. ; where each video frame Indicates the time point of data collection The image.

[0024] Step S102: For any video frame in the video frame sequence, calculate the inter-frame optical flow variation index of the video frame.

[0025] Specifically, the inter-frame optical flow variation index represents the degree of motion change between the video frame and the previous video frame.

[0026] In some specific embodiments, step S102 above includes steps S1021-S1024: Step S1021: For any pixel among multiple pixels in the video frame, calculate the optical flow vector of the pixel between the video frame and the previous video frame.

[0027] Specifically, the optical flow vector Represents the pixel point In two adjacent video frames (i.e., video frames) i Compared to the previous video frame i-1 The direction and distance of movement between ( ).

[0028] Step S1022: Calculate the Euclidean norm of the optical flow vector.

[0029] Specifically, for each optical flow vector, its Euclidean norm is calculated, i.e. The Euclidean norm (also known as the L2 norm) represents the motion amplitude of the pixel between two adjacent video frames.

[0030] Step S1023: Summing the multiple Euclidean norms corresponding one-to-one with the multiple pixels to obtain a summation result.

[0031] Specifically, for the first i For each pixel location in the frame, sum the L2 norm of the optical flow vectors of all pixels. That is, sum the values ​​of the optical flow vectors for the image width. and height All pixels within ,calculate:

[0032] Step S1024: Calculate the inter-frame optical flow variation index of the video frame based on the summation result and the total image area of ​​the video frame.

[0033] Specifically, the summation result can be divided by the number of video frames. i The total area of ​​the image is obtained from the video frame. i Inter-frame optical flow variation index.

[0034] In steps S1021-S1024 above, the video frame can be calculated using the following formula. i Inter-frame optical flow variation indicators:

[0035] in, Indicates the first Inter-frame optical flow variation index for each video frame; Indicates the first The total area of ​​the image in each video frame. Indicates the height of the image. Indicates the width of the image; Indicates the first The video frame and the first Pixels between video frames The optical flow vector; The Euclidean norm of the optical flow vector is denoted by .

[0036] Step S103: Divide the video frame sequence into a key frame set and a redundant frame set.

[0037] Specifically, the inter-frame optical flow variation index of the keyframes in the keyframe set is greater than or equal to the optical flow variation threshold, while the inter-frame optical flow variation index of the redundant frames in the redundant frame set is less than the optical flow variation threshold. The optical flow variation threshold can be set according to actual conditions and is not specifically limited here.

[0038] In some specific embodiments, step S103 above includes steps S1031-S1034: Step S1031: For any video frame in the video frame sequence, determine the relationship between the inter-frame optical flow change index and the optical flow change threshold of the video frame.

[0039] Step S1032: If the inter-frame optical flow change index of the video frame is greater than or equal to the optical flow change threshold, then the video frame is determined as a key frame.

[0040] Step S1033: If the inter-frame optical flow change index of the video frame is less than the optical flow change threshold, then the video frame is determined to be a redundant frame.

[0041] Step S1034: Establish the key frame set based on multiple key frames, and establish the redundant frame set based on multiple redundant frames.

[0042] In steps S1031-S1034 above, the keyframe set can be determined in the following way. K and redundant frame set R :

[0043] in, Represents the first frame in the video frame sequence. i One video frame, Indicates the first i Inter-frame optical flow variation index of each video frame Indicates the threshold for optical flow variation; In other words: when any video frame i Inter-frame optical flow variation index Greater than or equal to the optical flow change threshold Then the first i Each video frame was identified as a keyframe. Conversely, the same applies to any video frame. i Inter-frame optical flow variation index Less than the optical flow change threshold Then the first i One video frame was identified as a redundant frame, that is: Video frame sequence Except for keyframes All other video frames are redundant frames.

[0044] Step S104: Perform image encoding on each keyframe in the keyframe set to obtain an image bitstream.

[0045] Specifically, each frame in the keyframe set can be encoded with high quality using a video coding standard (such as H.264 / H.265) to generate an image bitstream. :

[0046] Step S105: Semantic text recognition is performed on each redundant frame in the redundant frame set using multiple recognition models, and a semantic text code stream is generated by combining the semantic text recognition results corresponding to the multiple redundant frames.

[0047] Specifically, the various recognition models include scene recognition models, object detection models, and text information extraction models; among them, scene recognition models include, but are not limited to, the RAM++ model, object detection models include, but are not limited to, the EVA02 model and the OWL-ViTv2 model, and text information extraction models include, but are not limited to, the PaddleOCR model and the LLaVA-NEXT model.

[0048] In some specific embodiments, step S105 above includes steps S1051-S1055: Step S1051: For any redundant frame, extract the scene label information of the redundant frame according to the scene recognition model.

[0049] Step S1052: Extract object information from the redundant frame according to the object detection model.

[0050] The object information includes object attribute information and object location information.

[0051] Step S1053: Extract the text information of the redundant frame according to the text information extraction model.

[0052] Specifically, there are no restrictions on the specific implementation methods of the above-mentioned scene recognition model, object detection model, and text information extraction model; as long as the corresponding functions of the respective models are implemented, for example: Suppose there is a video frame in a traffic scene. The scene recognition model can identify the scene label of the video frame as a city street; the object detection model can identify that there is a red car at one image location in the video frame and a blue bicycle at another image location in the video frame; the text information extraction model can identify the text information "No Parking" in the video frame.

[0053] Step S1054: Determine the timestamp information for semantic text recognition of the redundant frame.

[0054] Step S1055: The scene label information, the object information, the text information, and the timestamp information are merged to obtain the semantic text recognition result of the redundant frame.

[0055] Specifically, the semantic text recognition results obtained by merging are shown below:

[0056] in, Indicates based on the first i The semantic text recognition results obtained from the redundant frames Indicates scene label information, Represents object information, Represents text information. Represents timestamp information.

[0057] In step S105 above, the semantic text recognition result corresponding to each redundant frame can be encoded by the BPE word segmenter to obtain the encoding result; then the multiple encoding results corresponding to multiple redundant frames are merged to obtain the semantic text code stream.

[0058] As shown in the following formula:

[0059] in, Indicates the first i The encoding result of the redundant frames, This refers to the BPE word segmenter. Indicates the first i The semantic text recognition results corresponding to the redundant frames.

[0060] Step S106: Send the image stream and the semantic text stream to the cloud server.

[0061] Example 2: Corresponding to the above video encoding methods, this application also provides a video decoding method applied to a cloud server. Figure 2 This is a flowchart of a video decoding method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: Receive the image bitstream and the semantic text bitstream.

[0062] Specifically, the image bitstream is obtained by encoding each keyframe in the keyframe set. The keyframe set consists of video frames in the video frame sequence whose inter-frame optical flow change index is greater than or equal to the optical flow change threshold. The inter-frame optical flow change index represents the degree of motion change between the video frame and the previous video frame. The semantic text bitstream is generated based on multiple semantic text recognition results corresponding to multiple redundant frames. Each semantic text recognition result is obtained by performing semantic text recognition on the corresponding redundant frames in the redundant frame set using multiple recognition models. The redundant frame set consists of video frames in the video frame sequence whose inter-frame optical flow change index is less than the optical flow change threshold.

[0063] Step S202: Decode the image bitstream to obtain multiple keyframes, and decode the semantic text bitstream to obtain multiple semantic text recognition results.

[0064] Step S203: Combine the multiple keyframes and multiple semantic text recognition results to obtain reconstructed video sequence information.

[0065] Specifically, the reconstructed video sequence information can be obtained by combining multiple keyframes, the timestamp information corresponding to each keyframe, multiple semantic text recognition results, and the timestamp information corresponding to each semantic text recognition result.

[0066] In some specific embodiments, after obtaining the reconstructed video sequence information, the method further includes: The reconstructed video sequence information is input into the trained abnormal event recognition model, and the abnormal event recognition result is output.

[0067] In this application embodiment, the abnormal event recognition model includes, but is not limited to, behavior recognition model, traffic event detection model and long-term question answering model, so as to complete the video-level multimodal reasoning task. The cloud server can optimize the key frame determination strategy and expert model configuration on the edge side based on temporal semantic feedback.

[0068] Example 3: Corresponding to the above video encoding methods, this application also provides an intelligent video analysis device for real-time traffic incident detection. The device includes: End-side devices (such as smart cameras): collect real-time traffic scene video data and obtain raw video sequences.

[0069] Edge devices (such as edge computing nodes): Calculate inter-frame optical flow changes according to the formula to determine the keyframe set and redundant frame set; perform efficient video compression encoding (such as H.265) on the keyframe set to generate keyframe bitstream; deploy scene understanding (RAM++), object detection (EVA02), and text recognition (PaddleOCR) models to extract semantic information for the redundant frame set, add a timestamp field, and construct a structured semantic text bitstream using the BPE algorithm; combine the image bitstream and semantic text bitstream and send them to the cloud.

[0070] Cloud-based devices (such as cloud analytics servers): receive and decode keyframe images and structured semantic text to reconstruct video temporal information; utilize multimodal video understanding models (such as LLaVA) to detect traffic anomalies and output detection results.

[0071] Compared to existing technologies, the intelligent video analysis device for real-time traffic event detection provided in this application, under an edge-cloud collaborative architecture, integrates frame-level structured semantic compression and timestamp alignment mechanisms. It dynamically extracts high-value semantic information from redundant frames in the video by deploying multi-type perception expert models on the edge, and organizes this information into a structured text stream with temporal information, thereby replacing the transmission requirements of most redundant frame images. This method not only significantly compresses the overall video bitrate but also effectively preserves the semantic integrity and temporal structure of the video content, exhibiting good real-time performance and scalability. It is suitable for the semantic understanding needs of multimodal large models in temporal perception tasks such as video question answering, event analysis, and behavior recognition, significantly reducing the transmission pressure between the edge and cloud while ensuring understanding accuracy.

[0072] Example 4: Corresponding to the above implementation methods of video encoding, this application also provides a video encoding apparatus for executing the video encoding methods described in the above embodiments. For example... Figure 3 As shown, the video encoding device includes: The sequence acquisition module is used to acquire video frame sequences; The index calculation module is used to calculate the inter-frame optical flow change index for any video frame in the video frame sequence; the inter-frame optical flow change index represents the degree of motion change between the video frame and the previous video frame. The sequence segmentation module is used to divide the video frame sequence into a key frame set and a redundant frame set; the inter-frame optical flow change index of the key frames in the key frame set is greater than or equal to the optical flow change threshold, and the inter-frame optical flow change index of the redundant frames in the redundant frame set is less than the optical flow change threshold. The image bitstream generation module is used to perform image encoding on each keyframe in the keyframe set to obtain an image bitstream; The semantic text stream generation module is used to perform semantic text recognition on each redundant frame in the redundant frame set using multiple recognition models, and to generate a semantic text stream from the multiple semantic text recognition results corresponding to multiple redundant frames. The code stream sending module is used to send the image code stream and the semantic text code stream to the cloud server.

[0073] Optionally, the index calculation module is further configured to: calculate the optical flow vector of any pixel among multiple pixels in the video frame between the video frame and the previous video frame; the optical flow vector represents the direction and distance of movement of the pixel between two adjacent video frames; calculate the Euclidean norm of the optical flow vector; the Euclidean norm represents the amplitude of movement of the pixel between two adjacent video frames; sum the multiple Euclidean norms corresponding one-to-one with the multiple pixels to obtain a summation result; and calculate the inter-frame optical flow variation index of the video frame based on the summation result and the total image area of ​​the video frame.

[0074] Optionally, the sequence segmentation module is further configured to: for any video frame in the video frame sequence, determine the relationship between the inter-frame optical flow change index and the optical flow change threshold of the video frame; if the inter-frame optical flow change index of the video frame is greater than or equal to the optical flow change threshold, then determine the video frame as a key frame; if the inter-frame optical flow change index of the video frame is less than the optical flow change threshold, then determine the video frame as a redundant frame; establish the key frame set based on multiple key frames, and establish the redundant frame set based on multiple redundant frames.

[0075] Optionally, the multiple recognition models include a scene recognition model, an object detection model, and a text information extraction model; the semantic text code stream generation module is further configured to: for any redundant frame, extract scene label information of the redundant frame according to the scene recognition model; extract object information of the redundant frame according to the object detection model; the object information includes object attribute information and object location information; extract text information of the redundant frame according to the text information extraction model; determine the timestamp information for semantic text recognition of the redundant frame; and merge the scene label information, the object information, the text information, and the timestamp information to obtain the semantic text recognition result of the redundant frame.

[0076] The video encoding apparatus and the video encoding method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0077] This application also provides an electronic device for performing the above-described video encoding or video decoding method. Please refer to... Figure 4 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 4 As shown, the electronic device 4 includes: a processor 400, a memory 401, a bus 402, and a communication interface 404. The processor 400, the communication interface 404, and the memory 401 are connected via the bus 402. The memory 401 stores a computer program that can run on the processor 400. When the processor 400 runs the computer program, it executes the video encoding method or video decoding method provided in the foregoing embodiments of this application.

[0078] The memory 401 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 404 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0079] Bus 402 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 401 is used to store programs. After receiving an execution instruction, the processor 400 executes the program. The video encoding or video decoding methods disclosed in the foregoing embodiments can be applied to the processor 400, or implemented by the processor 400.

[0080] The processor 400 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 400 or by instructions in software form. The processor 400 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 401. The processor 400 reads the information in memory 401 and, in conjunction with its hardware, completes the steps of the above method.

[0081] The electronic device provided in this application embodiment and the video encoding method or video decoding method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0082] This application also provides a computer-readable storage medium corresponding to the video encoding or video decoding method provided in the foregoing embodiments. Please refer to... Figure 5 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the video encoding method or video decoding method provided in any of the foregoing embodiments.

[0083] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0084] The computer-readable storage medium provided in the above embodiments of this application and the video encoding method or video decoding method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0085] It should be noted that: Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0086] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0087] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0088] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video encoding method, characterized in that, The method includes: Obtain the video frame sequence; For any video frame in the video frame sequence, calculate the inter-frame optical flow variation index of the video frame; the inter-frame optical flow variation index represents the degree of motion change between the video frame and the previous video frame. The video frame sequence is divided into a key frame set and a redundant frame set; the inter-frame optical flow change index of the key frames in the key frame set is greater than or equal to the optical flow change threshold, and the inter-frame optical flow change index of the redundant frames in the redundant frame set is less than the optical flow change threshold. Each keyframe in the keyframe set is image encoded to obtain an image bitstream; Semantic text recognition is performed on each redundant frame in the redundant frame set using multiple recognition models, and a semantic text code stream is generated by combining the semantic text recognition results of multiple redundant frames. The image stream and the semantic text stream are sent to the cloud server; The multiple recognition models include a scene recognition model, an object detection model, and a text information extraction model; semantic text recognition is performed on each redundant frame in the redundant frame set using these multiple recognition models, including: For any redundant frame, the scene label information of the redundant frame is extracted according to the scene recognition model; The object information of the redundant frame is extracted according to the object detection model; the object information includes object attribute information and object location information. The text information of the redundant frames is extracted according to the text information extraction model; Determine the timestamp information for semantic text recognition of the redundant frames; The scene label information, the object information, the text information, and the timestamp information are combined to obtain the semantic text recognition result of the redundant frame.

2. The method according to claim 1, characterized in that, Calculating the inter-frame optical flow variation index of the video frame includes: For any pixel among multiple pixels in the video frame, calculate the optical flow vector of the pixel between the video frame and the previous video frame; the optical flow vector represents the direction and distance of the pixel's movement between two adjacent video frames. Calculate the Euclidean norm of the optical flow vector; the Euclidean norm represents the motion amplitude of the pixel between two adjacent video frames; The summation result is obtained by summing the multiple Euclidean norms that correspond one-to-one with the multiple pixels; Based on the summation result and the total area of ​​the video frame, the inter-frame optical flow variation index of the video frame is calculated.

3. The method according to claim 1 or 2, characterized in that, The video frame sequence is divided into a keyframe set and a redundant frame set, including: For any video frame in the video frame sequence, determine the relationship between the inter-frame optical flow change index and the optical flow change threshold of the video frame; If the inter-frame optical flow change index of the video frame is greater than or equal to the optical flow change threshold, then the video frame is determined as a key frame. If the inter-frame optical flow variation index of the video frame is less than the optical flow variation threshold, then the video frame is determined to be a redundant frame. The keyframe set is established based on multiple keyframes, and the redundant frame set is established based on multiple redundant frames.

4. The method according to claim 1 or 2, characterized in that, Calculating the inter-frame optical flow variation index of the video frame includes: in, Indicates the first Inter-frame optical flow variation index for each video frame; Indicates the first The total area of ​​the image in each video frame. Indicates the height of the image. Indicates the width of the image; Indicates the first The video frame and the first Pixels between video frames The optical flow vector; The Euclidean norm of the optical flow vector is denoted by .

5. A video decoding method, characterized in that, The method includes: The system receives an image stream and a semantic text stream. The image stream is obtained by encoding each keyframe in a keyframe set. The keyframe set consists of video frames in a video frame sequence whose inter-frame optical flow change index is greater than or equal to an optical flow change threshold. The inter-frame optical flow change index represents the degree of motion change between the current video frame and the previous video frame. The semantic text stream is generated based on multiple semantic text recognition results corresponding to multiple redundant frames. Each semantic text recognition result is obtained by performing semantic text recognition on the corresponding redundant frames in the redundant frame set using multiple recognition models. The redundant frame set consists of video frames in a video frame sequence whose inter-frame optical flow change index is less than the optical flow change threshold. The image stream is decoded to obtain multiple keyframes, and the semantic text stream is decoded to obtain multiple semantic text recognition results; The reconstructed video sequence information is obtained by combining the multiple keyframes and multiple semantic text recognition results; The various recognition models include scene recognition models, object detection models, and text information extraction models; semantic text recognition is performed on corresponding redundant frames in the redundant frame set using these various recognition models, including: For any redundant frame, the scene label information of the redundant frame is extracted according to the scene recognition model; The object information of the redundant frame is extracted according to the object detection model; the object information includes object attribute information and object location information. The text information of the redundant frames is extracted according to the text information extraction model; Determine the timestamp information for semantic text recognition of the redundant frames; The scene label information, the object information, the text information, and the timestamp information are combined to obtain the semantic text recognition result of the redundant frame.

6. The method according to claim 5, characterized in that, After obtaining the reconstructed video sequence information, the method further includes: The reconstructed video sequence information is input into the trained abnormal event recognition model, and the abnormal event recognition result is output.

7. A video encoding device, characterized in that, The device includes: The sequence acquisition module is used to acquire video frame sequences; The index calculation module is used to calculate the inter-frame optical flow change index for any video frame in the video frame sequence; the inter-frame optical flow change index represents the degree of motion change between the video frame and the previous video frame. The sequence segmentation module is used to divide the video frame sequence into a key frame set and a redundant frame set; the inter-frame optical flow change index of the key frames in the key frame set is greater than or equal to the optical flow change threshold, and the inter-frame optical flow change index of the redundant frames in the redundant frame set is less than the optical flow change threshold. The image bitstream generation module is used to perform image encoding on each keyframe in the keyframe set to obtain an image bitstream; The semantic text stream generation module is used to perform semantic text recognition on each redundant frame in the redundant frame set using multiple recognition models, and to generate a semantic text stream from the multiple semantic text recognition results corresponding to multiple redundant frames. The code stream sending module is used to send the image code stream and the semantic text code stream to the cloud server; The multiple recognition models include a scene recognition model, an object detection model, and a text information extraction model; semantic text recognition is performed on each redundant frame in the redundant frame set using these multiple recognition models, including: For any redundant frame, the scene label information of the redundant frame is extracted according to the scene recognition model; The object information of the redundant frame is extracted according to the object detection model; the object information includes object attribute information and object location information. The text information of the redundant frames is extracted according to the text information extraction model; Determine the timestamp information for semantic text recognition of the redundant frames; The scene label information, the object information, the text information, and the timestamp information are combined to obtain the semantic text recognition result of the redundant frame.

8. A computer device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 6.

Images