Video encoding method, video decoding method, and encoding end and / or decoding end

Abstract

The present invention relates to the field of video processing, and in particular to a video encoding method, a video decoding method, and an encoding end or / and a decoding end. The method comprises: monitoring input events; when the input events are monitored, encoding and processing the current video data frame to generate encoding data; reconstructing the encoding data to generate a reconstructed video data frame; based on the difference between the current video data frame and the reconstructed video data frame, generating additional information corresponding to the current video data frame; the additional information is used to characterize identification information that needs to be displayed in the current video data frame; transmitting the encoding data and the additional information to a decoding end corresponding to the encoding end. The above-mentioned video encoding method not only generates encoding data, but also generates additional information, so that after the decoding end decodes the encoding data and the additional information, it can ensure that not only the decoded video frame is clear, but also the identification information in the decoded video frame is clear.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a video encoding method, a video decoding method, and an encoding end and / or a decoding end. Background Art

[0002] With the gradual popularization of 5G, cloud computing is increasingly being used. Cloud computing products have penetrated into all walks of life in hundreds of cities across the country. The terminals used by customers, application scenarios, and shared public network conditions vary greatly. Currently, cloud gaming is an important part of the cloud computing product matrix for the majority of individual users.

[0003] The cloud gaming industry has unique application scenarios, which are characterized by rapid and continuous image changes, full image refresh, rich color information, and high bandwidth requirements. The current transmission method is to compress and transmit video streams. The purpose is to utilize the characteristics of video stream encoders that are good at compressing time and space redundancy, and to compress the amount of data that needs to be transmitted and displayed to the greatest extent while maintaining visual fluency, so as to reduce the bandwidth requirements of cloud gaming and save bandwidth overhead.

[0004] However, when the video encoder compresses the video stream, some parameters are discarded, which affects the image edges. In frequently used interactive input scenarios, the edges of the text will not be sharp enough, making the text in the video unclear. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a video encoding method, aiming to solve the problem of unclear text in a video.

[0006] According to a first aspect, an embodiment of the present invention provides a video encoding method, which is applied to an encoding end; comprising:

[0007] Listen for input events;

[0008] When an input event is monitored, the current video data frame is encoded to generate encoded data;

[0009] Reconstructing the encoded data to generate a reconstructed video data frame;

[0010] Based on the difference between the current video data frame and the reconstructed video data frame, additional information corresponding to the current video data frame is generated; the additional information is used to represent identification information that needs to be displayed in the current video data frame;

[0011] The encoded data and additional information are transmitted to the decoding end corresponding to the encoding end.

[0012] The video encoding method provided by the embodiment of the present invention monitors the input event. When the input event is monitored, the current video data frame is encoded and processed to generate the encoded data, thereby ensuring the accuracy of the generated encoded data. Then, the encoded data is reconstructed to generate a reconstructed video data frame, thereby ensuring the accuracy of the generated reconstructed video data frame. Based on the difference between the current video data frame and the reconstructed video data frame, additional information corresponding to the current video data frame is generated, wherein the additional information is used to characterize the identification information that needs to be displayed in the current video data frame; thereby ensuring the accuracy of the generated additional information. The encoded data and the additional information are transmitted to the decoding end corresponding to the encoding end. The above-mentioned video encoding method not only encodes the current video data frame to generate the encoded data, but also generates the additional information corresponding to the identification information that needs to be displayed in the current video data frame, thereby ensuring that after the decoding end decodes the encoded data and the additional information, not only the decoded video frame is clear, but also the identification information in the decoded video frame is clear.

[0013] In combination with the first aspect, in a first implementation of the first aspect, generating additional information corresponding to the current video data frame based on a difference between the current video data frame and the reconstructed video data frame includes:

[0014] Subtracting the reconstructed video data frame from the current video data frame to obtain a first image residual;

[0015] Performing horizontal difference residual processing on the current video data frame to obtain a horizontal difference matrix;

[0016] Performing longitudinal difference residual processing on the current video data frame to obtain a longitudinal difference matrix;

[0017] Generate a second image residual based on the relationship between the horizontal difference matrix and the vertical difference matrix;

[0018] Based on the relationship between the first image residual and the second image residual, generating a target residual matrix;

[0019] The target residual matrix is ​​analyzed to generate additional information.

[0020] The video encoding method provided by the embodiment of the present invention uses the current video data frame to subtract the reconstructed video data frame to obtain the first image residual, thereby ensuring the accuracy of the generated first image residual. Then, the current video data frame is subjected to horizontal differential residual processing to obtain a horizontal differential matrix; the current video data frame is subjected to vertical differential residual processing to obtain a vertical differential matrix, thereby ensuring the accuracy of the generated horizontal differential matrix and the vertical differential matrix. Based on the relationship between the horizontal differential matrix and the vertical differential matrix, the second image residual is generated, thereby ensuring the accuracy of the generated second image residual. Based on the relationship between the first image residual and the second image residual, the target residual matrix is ​​generated, thereby ensuring the accuracy of the generated target residual matrix. Then, the target residual matrix is ​​analyzed to generate additional information, thereby ensuring the accuracy of the generated additional information, thereby ensuring that the identification information in the decoded video frame is clear.

[0021] In combination with the first implementation of the first aspect, in the second implementation of the first aspect, based on the relationship between the horizontal difference matrix and the vertical difference matrix, generating the second image residual includes:

[0022] The corresponding elements in the horizontal difference matrix and the vertical difference matrix are squared, the squared data are summed, and the square root of the summed value is calculated to generate the second image residual.

[0023] The video encoding method provided by the embodiment of the present invention squares the corresponding elements in the horizontal difference matrix and the vertical difference matrix, sums the squared data, and performs square root calculation on the summed value to generate a second image residual, thereby ensuring the accuracy of the generated second image residual and further ensuring the accuracy of the generated additional information.

[0024] In combination with the first implementation of the first aspect, in a third implementation of the first aspect, based on the relationship between the first image residual and the second image residual, a target residual matrix is ​​generated, including:

[0025] Obtaining a first preset threshold corresponding to the second image residual;

[0026] Comparing each element in the second image residual with a first preset threshold;

[0027] According to the comparison result, the second image residual is changed into a binary matrix;

[0028] The binary matrix is ​​ANDed with the first image residual to generate a target residual matrix.

[0029] The video encoding method provided by the embodiment of the present invention obtains the first preset threshold corresponding to the second image residual, and then compares each element in the second image residual with the first preset threshold, thereby ensuring the accuracy of the obtained comparison result. Then, according to the comparison result, the second image residual is changed into a binary matrix, thereby ensuring the accuracy of the generated binary matrix. The binary matrix is ​​ANDed with the first image residual to generate a target residual matrix, thereby ensuring the accuracy of the generated target residual matrix, and further ensuring the accuracy of the generated additional information.

[0030] In combination with the first implementation of the first aspect, in a fourth implementation of the first aspect, the target residual matrix is ​​analyzed to generate additional information, including:

[0031] Obtaining a second preset threshold corresponding to the target residual matrix;

[0032] Based on the relationship between the second preset threshold and each element in the target residual matrix, thresholding the target residual matrix to generate first candidate additional information;

[0033] Performing DCT transformation on the first candidate additional information to generate second candidate additional information;

[0034] The second candidate additional information is scanned and encoded to generate additional information.

[0035] The video encoding method provided by the embodiment of the present invention obtains a second preset threshold corresponding to the target residual matrix, performs thresholding processing on the target residual matrix based on the relationship between the second preset threshold and each element in the target residual matrix, generates first candidate additional information, and ensures the accuracy of the generated first candidate additional information. Then, DCT conversion is performed on the first candidate additional information to generate second candidate additional information, and the accuracy of the generated second candidate additional information is ensured. The second candidate additional information is scanned and encoded to generate additional information, and the accuracy of the generated additional information is ensured.

[0036] In combination with the first aspect, in a fifth implementation of the first aspect, transmitting the encoded data and the additional information to a decoding end corresponding to the encoding end includes:

[0037] Analyze the additional information and generate identification information corresponding to the additional information;

[0038] The encoded data, the additional information, and the identification information corresponding to the additional information are transmitted to a decoding end corresponding to the encoding end.

[0039] The video encoding method provided by the embodiment of the present invention analyzes the additional information and generates identification information corresponding to the additional information; the encoded data, the additional information and the identification information corresponding to the additional information are transmitted to the decoding end corresponding to the encoding end, thereby ensuring that the decoding end can accurately determine whether there is an additional message based on the identification information corresponding to the additional information, and there is no need to decode and read the additional information before determining whether there is additional information, thereby reducing the workload of the decoding end and improving the working efficiency of the decoding end.

[0040] According to a second aspect, an embodiment of the present invention further provides a video decoding method, which is applied to a decoding end; comprising:

[0041] Receiving coded data and additional information sent by a coding end, where the coded data and additional information are generated according to the video coding method of the first aspect and any one of the implementation methods of the first aspect;

[0042] Decoding and reconstructing the encoded data to generate a reconstructed video data frame;

[0043] Perform inverse DCT transformation on the additional information to generate a target additional matrix;

[0044] Based on the relationship between the target additional matrix and the reconstructed video data frame, the reconstructed video data frame is optimized to generate a target reconstructed video frame;

[0045] The target reconstructed video frame is superimposed with the target additional matrix and rendered for output display.

[0046] The video decoding method provided by the embodiment of the present invention receives the coded data and additional information sent by the coding end, then decodes and reconstructs the coded data to generate a reconstructed video data frame, thereby ensuring the accuracy of the generated reconstructed video data frame. Then, the additional information is subjected to an inverse DCT transform to generate a target additional matrix, thereby ensuring the accuracy of the generated target additional matrix. Based on the relationship between the target additional matrix and the reconstructed video data frame, the reconstructed video data frame is optimized to generate a target reconstructed video frame, thereby ensuring the accuracy of the generated target reconstructed video frame. The target reconstructed video frame is superimposed on the target additional matrix and rendered for output display, thereby ensuring the clarity of the output displayed target reconstructed video frame and the clarity of the identification information in the target reconstructed video frame.

[0047] In combination with the second aspect, in a first implementation of the second aspect, based on the relationship between the target additional matrix and the reconstructed video data frame, the reconstructed video data frame is optimized to generate a target reconstructed video frame.

[0048] Performing horizontal and vertical differential processing on the reconstructed video data frame, and generating candidate image residuals based on the relationship between the generated matrices;

[0049] Based on the difference between the candidate image residual and the target additional matrix, a target optimization function is constructed;

[0050] The target optimization function is solved, and according to the solution result, the reconstructed video data frame is updated to generate the target reconstructed video frame.

[0051] The video decoding method provided by the embodiment of the present invention performs horizontal differential processing and vertical differential processing on the reconstructed video data frame, and generates a candidate image residual based on the relationship between the generated matrices, thereby ensuring the accuracy of the generated candidate image residual. Based on the difference between the candidate image residual and the target additional matrix, a target optimization function is constructed, thereby ensuring the accuracy of the constructed target optimization function. Then, the target optimization function is solved, and according to the solution result, the reconstructed video data frame is updated to generate a target reconstructed video frame, thereby ensuring the accuracy of the generated target reconstructed video frame.

[0052] According to a third aspect, an embodiment of the present invention further provides an image encoding device, which is applied to an encoding end; comprising:

[0053] The monitoring module is used to monitor input events.

[0054] The encoding process is used to encode the current video data frame to generate encoded data when an input event is monitored.

[0055] The first reconstruction module is used to reconstruct the encoded data to generate a reconstructed video data frame.

[0056] The generation module is used to generate additional information corresponding to the current video data frame based on the difference between the current video data frame and the reconstructed video data frame; the additional information is used to represent identification information that needs to be displayed in the current video data frame.

[0057] The transmission module is used to transmit the encoded data and additional information to the decoding end corresponding to the encoding end.

[0058] The image encoding device provided by the embodiment of the present invention monitors the input event. When the input event is monitored, the current video data frame is encoded and processed to generate the encoded data, thereby ensuring the accuracy of the generated encoded data. Then, the encoded data is reconstructed to generate a reconstructed video data frame, thereby ensuring the accuracy of the generated reconstructed video data frame. Based on the difference between the current video data frame and the reconstructed video data frame, additional information corresponding to the current video data frame is generated, wherein the additional information is used to characterize the identification information that needs to be displayed in the current video data frame; thereby ensuring the accuracy of the generated additional information. The encoded data and the additional information are transmitted to the decoding end corresponding to the encoding end. The above-mentioned image encoding device not only encodes the current video data frame to generate the encoded data, but also generates the additional information corresponding to the identification information that needs to be displayed in the current video data frame, so that after the decoding end decodes the encoded data and the additional information, it can be ensured that not only the decoded video frame is clear, but also the identification information in the decoded video frame is clear.

[0059] In combination with the third aspect, in the first embodiment of the third aspect, the above-mentioned generation module is specifically used to: subtract the reconstructed video data frame from the current video data frame to obtain a first image residual; perform horizontal difference residual processing on the current video data frame to obtain a horizontal difference matrix; perform vertical difference residual processing on the current video data frame to obtain a vertical difference matrix; generate a second image residual based on the relationship between the horizontal difference matrix and the vertical difference matrix; generate a target residual matrix based on the relationship between the first image residual and the second image residual; analyze the target residual matrix to generate additional information.

[0060] In combination with the first implementation manner of the third aspect, in the second embodiment of the third aspect, the above-mentioned generation module is specifically used to: square the corresponding elements in the horizontal difference matrix and the vertical difference matrix, sum the squared data, and perform square root calculation on the summed value to generate a second image residual.

[0061] In combination with the first implementation manner of the third aspect, in the second embodiment of the third aspect, the above-mentioned generation module is specifically used to: obtain a first preset threshold corresponding to the second image residual; compare each element in the second image residual with the first preset threshold; according to the comparison result, change the second image residual into a binary matrix; perform an AND operation on the binary matrix and the first image residual to generate a target residual matrix.

[0062] In combination with the first implementation manner of the third aspect, in the second embodiment of the third aspect, the above-mentioned generation module is specifically used to: obtain a second preset threshold corresponding to the target residual matrix; based on the relationship between the second preset threshold and each element in the target residual matrix, perform thresholding processing on the target residual matrix to generate first candidate additional information; perform DCT conversion on the first candidate additional information to generate second candidate additional information; scan and encode the second candidate additional information to generate additional information.

[0063] In combination with the first implementation manner of the third aspect, in the second embodiment of the third aspect, the above-mentioned transmission module is specifically used to: analyze the additional information and generate identification information corresponding to the additional information; transmit the encoded data, additional information and the identification information corresponding to the additional information to the decoding end corresponding to the encoding end.

[0064] According to a fourth aspect, an embodiment of the present invention further provides an image decoding device, applied to a decoding end; comprising:

[0065] The receiving module is used to receive the encoded data and additional information sent by the encoding end, where the encoded data and additional information are generated according to the video encoding method of the first aspect and any one of the implementation methods of the first aspect.

[0066] The second reconstruction module is used to decode and reconstruct the encoded data to generate a reconstructed video data frame.

[0067] An inverse transform module, used for performing DCT inverse transform on the additional information to generate a target additional matrix;

[0068] The optimization module is used to optimize the reconstructed video data frame based on the relationship between the target additional matrix and the reconstructed video data frame to generate a target reconstructed video frame.

[0069] The overlay module is used to overlay the target reconstructed video frame with the target additional matrix and render it for output display.

[0070] The image decoding device provided by the embodiment of the present invention receives the coded data and additional information sent by the coding end, then decodes and reconstructs the coded data to generate a reconstructed video data frame, thereby ensuring the accuracy of the generated reconstructed video data frame. Then, the additional information is subjected to an inverse DCT transform to generate a target additional matrix, thereby ensuring the accuracy of the generated target additional matrix. Based on the relationship between the target additional matrix and the reconstructed video data frame, the reconstructed video data frame is optimized to generate a target reconstructed video frame, thereby ensuring the accuracy of the generated target reconstructed video frame. The target reconstructed video frame is superimposed on the target additional matrix and rendered for output display, thereby ensuring the clarity of the output displayed target reconstructed video frame and the clarity of the identification information in the target reconstructed video frame.

[0071] According to the fifth aspect, an embodiment of the present invention provides an encoding end and / or decoding end, including a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the video encoding method in the first aspect or any one of the embodiments of the first aspect and the video decoding method in the second aspect or any one of the embodiments of the second aspect by executing the computer instructions.

[0072] According to the sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the video encoding method in the first aspect or any one of the embodiments of the first aspect and the video decoding method in the second aspect or any one of the embodiments of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 is a flow chart of a video encoding method provided by an embodiment of the present invention;

[0075] Figure 2 is a flowchart of a video encoding method provided by another embodiment of the present invention;

[0076] Figure 3 is a schematic diagram of generating a target residual matrix provided by another embodiment of the present invention;

[0077] Figure 4 is a schematic diagram of a process for generating additional information provided by another embodiment of the present invention;

[0078] Figure 5 is a flowchart of a video encoding method provided by another embodiment of the present invention;

[0079] Figure 6 is a flowchart of a video decoding method provided by another embodiment of the present invention;

[0080] Figure 7 is a schematic diagram of applying a decoding end provided by another embodiment of the present invention to decode coded data and additional information;

[0081] Figure 8It is a schematic diagram of the whole process of encoding and decoding video frame data provided by another embodiment of the present invention;

[0082] Fig. 9 is a functional module diagram of an image encoding device provided by an embodiment of the present invention;

[0083] Fig.10 is a functional module diagram of an image decoding device provided by an embodiment of the present invention;

[0084] Fig.11 It is a schematic diagram of the hardware structure of the encoding end and / or the decoding end provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0086] It should be noted that the image coding method provided in the embodiment of the present application may be executed by an image coding device, which may be implemented as part or all of the coding end device through software, hardware, or a combination of software and hardware, wherein the coding end may be a server or a terminal, wherein the server in the embodiment of the present application may be a single server or a server cluster composed of multiple servers, and the terminal in the embodiment of the present application may be a smart phone, a personal computer, a tablet computer, a wearable device, an intelligent robot, or other intelligent hardware devices. In the following method embodiments, the execution subject is taken as an example for explanation.

[0087] In one embodiment of the present application, Figure 1 As shown, a video encoding method is provided, which is described by taking the method applied to an encoding end as an example, and includes the following steps:

[0088] S11. Monitor input events.

[0089] Specifically, a virtual machine input event listener is provided on the virtual machine side of the encoding end, and the encoding end can monitor input events based on the virtual machine input event listener.

[0090] The input event may be a user inputting text, letters, characters, and other identification information based on an input component, and the input component may be a keyboard or a mouse, etc. The implementation of this application does not specifically limit the input event.

[0091] S12: When an input event is monitored, the current video data frame is encoded to generate encoded data.

[0092] Specifically, when the encoding end monitors an input event, the encoding end can obtain the current video data frame, and use the video stream encoder to encode the current video data frame according to the content of the current video data frame and the relationship with the previous and next frames to generate encoded data.

[0093] Among them, the video stream encoding method used by the video stream encoder can be any one of the H.264 encoding method, MPEG-4 encoding method, MPEG-2TS encoding method, WMV-HD encoding method and VC-1 encoding method. The embodiment of the present application does not specifically limit the video stream encoding method used by the video stream encoder.

[0094] In an optional implementation of the present application, when the encoding end does not monitor an input event, the encoding end may send the encoded data to the decoding end after generating the encoded data, so that the decoding end may decode the encoded data and display it after rendering.

[0095] S13: Reconstruct the encoded data to generate a reconstructed video data frame.

[0096] Specifically, after the encoding end performs encoding processing on the current video data frame to generate encoded data, the encoding end may reconstruct the encoded data to generate a reconstructed video data frame.

[0097] Optionally, the encoding end may use the reconstructed video data frame as a reference video data frame, so as to compare the reconstructed video data frame with the current video data frame.

[0098] S14. Generate additional information corresponding to the current video data frame based on the difference between the current video data frame and the reconstructed video data frame.

[0099] The additional information is used to represent identification information that needs to be displayed in the current video data frame. The identification information can be text information, letter information, number information, icon information, etc. The embodiment of the present application does not specifically limit the additional information.

[0100] Specifically, when the encoder detects an input event, in order to retain more detailed information of the current video data frame and reduce the loss of the image edge of the current video data frame, the encoder needs to generate additional information corresponding to the current video data frame and use the additional information as the basis for reconstructing the current video data frame.

[0101] In an optional implementation of the present application, the encoding end can use the current video data frame to subtract the reconstructed video data frame to obtain the difference between the current video data frame and the reconstructed video data frame, and generate additional information corresponding to the current video data frame based on the difference between the current video data frame and the reconstructed video data frame.

[0102] This step will be described in detail below.

[0103] S15, transmitting the encoded data and the additional information to the decoding end corresponding to the encoding end.

[0104] Specifically, after generating the additional information, the encoding end may transmit the encoded data and the additional information to the decoding end corresponding to the encoding end based on the communication connection between the encoding end and the decoding end.

[0105] In an optional implementation of the present application, from the frequency of the input event monitoring information, it is actually a discrete time series, and the interval time is much greater than the general decoder frame rate (33ms, 30fps), so the input event message needs to be considered valid within a time period after being monitored, and the video data frames within this range must be encoded and decoded in a manner with additional information. From the current debugging point of view, this time range is generally set to 2s, that is, when the input event is monitored, at least 60 frames of video data frames thereafter must be encoded and decoded in the above manner. If the input event is updated again during this period, it will be deferred in sequence until interrupted to ensure a consistent visual effect and user experience in the text input scenario.

[0106] The video encoding method provided by the embodiment of the present invention monitors the input event. When the input event is monitored, the current video data frame is encoded and processed to generate the encoded data, thereby ensuring the accuracy of the generated encoded data. Then, the encoded data is reconstructed to generate a reconstructed video data frame, thereby ensuring the accuracy of the generated reconstructed video data frame. Based on the difference between the current video data frame and the reconstructed video data frame, additional information corresponding to the current video data frame is generated, wherein the additional information is used to characterize the identification information that needs to be displayed in the current video data frame; thereby ensuring the accuracy of the generated additional information. The encoded data and the additional information are transmitted to the decoding end corresponding to the encoding end. The above-mentioned video encoding method not only encodes the current video data frame to generate the encoded data, but also generates the additional information corresponding to the identification information that needs to be displayed in the current video data frame, thereby ensuring that after the decoding end decodes the encoded data and the additional information, not only the decoded video frame is clear, but also the identification information in the decoded video frame is clear.

[0107] In one embodiment of the present application, Figure 2As shown, a video encoding method is provided, which is described by taking the method applied to an encoding end as an example, and includes the following steps:

[0108] S21. Monitor input events.

[0109] For more information about this step, see Figure 1 The introduction of S11 will not be elaborated here.

[0110] S22: When an input event is monitored, the current video data frame is encoded to generate encoded data.

[0111] For more information about this step, see Figure 1 The introduction of S12 will not be elaborated here.

[0112] S23: reconstruct the encoded data to generate a reconstructed video data frame.

[0113] For more information about this step, see Figure 1 The introduction of S13 will not be elaborated here.

[0114] S24: Generate additional information corresponding to the current video data frame based on the difference between the current video data frame and the reconstructed video data frame.

[0115] The additional information is used to represent identification information that needs to be displayed in the current video data frame.

[0116] In an optional implementation manner of the present application, the above S24 “generating additional information corresponding to the current video data frame based on the difference between the current video data frame and the reconstructed video data frame” may include the following steps:

[0117] S241. Subtract the reconstructed video data frame from the current video data frame to obtain a first image residual.

[0118] Specifically, when the encoding end monitors the input event, assuming that the current video data frame at the current moment is T, the encoding end sends the current video data frame to the stream encoder to encode the current video data frame. The stream encoder creates a cache reference queue consisting of reconstructed video data frames at the encoding end. When the current video data frame has been encoded and reconstructed into a reconstructed video data frame Tr, the reconstructed video data frame Tr is found in the cache reference queue. The encoding end can use the current video data frame T to subtract the reconstructed video data frame Tr to obtain a first image residual.

[0119] S242, performing horizontal difference residual processing on the current video data frame to obtain a horizontal difference matrix.

[0120] Specifically, after generating the first image residual, the encoding end may perform horizontal difference residual processing on the current video data frame to obtain a horizontal difference matrix.

[0121] Exemplarily, the encoding end may subtract the previous row of elements from the next row of elements in the current video data frame, and perform zero padding on the last row, thereby generating a horizontal difference matrix.

[0122] S243, performing longitudinal difference residual processing on the current video data frame to obtain a longitudinal difference matrix.

[0123] Specifically, after generating the horizontal difference matrix, the encoding end may perform vertical difference residual processing on the current video data frame to obtain the vertical difference matrix.

[0124] Exemplarily, the encoding end may subtract the row elements of the previous column from the elements of the next column in the current video data frame, and perform zero padding on the last column, thereby generating a vertical difference matrix.

[0125] S244. Generate a second image residual based on the relationship between the horizontal difference matrix and the vertical difference matrix.

[0126] In an optional implementation of the present application, the encoding end may use a horizontal difference matrix plus a vertical difference matrix to generate a second image residual.

[0127] In another optional implementation manner of the present application, the above S244 “generating a second image residual based on the relationship between the horizontal difference matrix and the vertical difference matrix” may include the following steps:

[0128] (1) Perform square calculation on the corresponding elements in the horizontal difference matrix and the vertical difference matrix, sum the squared data, and perform square root calculation on the summed value to generate the second image residual.

[0129] Specifically, the encoding end may perform square calculation on corresponding elements in the horizontal difference matrix and the vertical difference matrix, sum the squared data, and perform square root calculation on the summed value to generate the second image residual.

[0130] Exemplarily, assuming that the horizontal difference matrix and the vertical difference matrix are both 2×2 matrices, the encoding can be performed by squaring the elements in the first row and the first column of the horizontal difference matrix and the vertical difference matrix respectively, and then summing the data obtained after the square calculation, and then performing square root calculation on the summed data to generate the elements in the first row and the first column of the second image residual, and then calculating the values ​​of other elements in the second image residual in sequence to generate the second image residual.

[0131] S245. Generate a target residual matrix based on the relationship between the first image residual and the second image residual.

[0132] In an optional implementation of the present application, the encoding end may perform an AND operation on the first image residual and the second image residual to generate a target residual matrix.

[0133] In an optional implementation manner of the present application, the above S245 “generating a target residual matrix based on the relationship between the first image residual and the second image residual” may include the following steps:

[0134] (1) Obtain a first preset threshold corresponding to the second image residual.

[0135] (2) Compare each element in the second image residual with the first preset threshold.

[0136] (3) According to the comparison result, the second image residual is changed into a binary matrix.

[0137] (4) Perform an AND operation on the binary matrix and the first image residual to generate a target residual matrix.

[0138] Specifically, the encoding end can receive the first preset threshold corresponding to the second image residual input by the user, or receive the first preset threshold corresponding to the second image residual sent by other devices, and can also analyze the second image residual and generate the first preset threshold corresponding to the second image residual according to the size of each element in the second image residual.

[0139] After obtaining the first preset threshold corresponding to the second image residual, the encoding end may compare each element in the second image residual with the first preset threshold, and change the second image residual into a binary matrix according to the comparison result.

[0140] Exemplarily, the encoding end may compare each element in the second image residual with a first preset threshold value. When an element in the second image residual is greater than or equal to the first preset threshold value, the element in the second image residual is changed to 1; when the element in the second image residual is less than the first preset threshold value, the element in the second image residual is changed to 0, thereby changing the second image residual to a binary matrix according to the comparison result.

[0141] After generating the binary matrix, the encoder can perform an AND operation on the binary matrix and the first image residual to generate a target residual matrix. At this time, the target residual matrix retains edge information to the greatest extent.

[0142] like Figure 3As shown, the process of generating a target residual matrix is ​​shown, performing horizontal differential residual processing on the current video data frame to obtain a horizontal differential matrix; performing vertical differential residual processing on the current video data frame to obtain a vertical differential matrix; performing square calculations on the corresponding elements in the horizontal differential matrix and the vertical differential matrix, and summing the squared data, performing square root calculations on the summed values ​​to generate a second image residual, and then, based on the relationship between the first image residual and the second image residual, generating a target residual matrix.

[0143] S246. Analyze the target residual matrix to generate additional information.

[0144] In an optional implementation manner of the present application, the above S246 “analyzing the target residual matrix to generate additional information” may include the following steps:

[0145] (1) Obtain a second preset threshold corresponding to the target residual matrix.

[0146] (2) Based on the relationship between the second preset threshold and each element in the target residual matrix, threshold processing is performed on the target residual matrix to generate the first candidate additional information.

[0147] (3) Perform DCT transformation on the first candidate additional information to generate second candidate additional information.

[0148] (4) Scan and encode the second candidate additional information to generate additional information.

[0149] Specifically, the encoding end may receive a second preset threshold corresponding to the target residual matrix input by a user, or may receive a second preset threshold corresponding to the target residual matrix sent by other devices, or may analyze the target residual matrix, and generate the second preset threshold corresponding to the target residual matrix according to the size of each element in the target residual matrix. The embodiment of the present application does not specifically limit the manner in which the encoding end obtains the second preset threshold corresponding to the target residual matrix.

[0150] Then, the encoder may compare the second preset threshold with each element in the target residual matrix, perform thresholding on the target residual matrix according to the comparison result, and generate the first candidate additional information, thereby filtering out small interference in the target residual matrix and generating the first candidate additional information.

[0151] Then, the encoder applies DCT transform to move the energy of the first candidate additional information to the upper left corner. At this time, the first candidate additional information is thresholded again, and the frequency point signals not concentrated in the upper left corner area are set to zero, so as to further suppress the interference caused by stream coding and concentrate the residual energy, thereby generating the second candidate additional information.

[0152] After the encoder uses ZigZag scanning, it uses Huffman coding to scan and encode the second candidate additional information to generate additional information.

[0153] For example, Figure 4 As shown, a flow chart for generating additional information is shown. Specifically, when the encoder detects an input event, the current video data is input into the video stream encoder, and the video stream encoder is used to encode the current video data frame according to the content of the current video data frame and the relationship with the previous and next frames to generate encoded data. The encoder reconstructs the encoded data to generate a reconstructed video data frame. Then, based on the current video data frame minus the reconstructed video data frame, a first image residual is obtained, and a horizontal difference residual processing is performed on the current video data frame to obtain a horizontal difference matrix, and a vertical difference residual processing is performed on the current video data frame to obtain a vertical difference matrix. Then, the encoder generates a second image residual based on the relationship between the horizontal difference matrix and the vertical difference matrix; and generates a target residual matrix based on the relationship between the first image residual and the second image residual. A second preset threshold corresponding to the target residual matrix is ​​obtained; based on the relationship between the second preset threshold and each element in the target residual matrix, the target residual matrix is ​​thresholded to generate a first candidate additional information; the first candidate additional information is DCT converted to generate a second candidate additional information; the second candidate additional information is scanned and encoded to generate additional information.

[0154] S25, transmitting the encoded data and the additional information to the decoding end corresponding to the encoding end.

[0155] For more information about this step, see Figure 1 The introduction of S15 will not be elaborated here.

[0156] The video encoding method provided by the embodiment of the present invention uses the current video data frame to subtract the reconstructed video data frame to obtain the first image residual, thereby ensuring the accuracy of the generated first image residual. Then, the current video data frame is subjected to horizontal differential residual processing to obtain a horizontal differential matrix; the current video data frame is subjected to vertical differential residual processing to obtain a vertical differential matrix, thereby ensuring the accuracy of the generated horizontal differential matrix and the vertical differential matrix. The corresponding elements in the horizontal differential matrix and the vertical differential matrix are squared, and the squared data are summed, and the square root of the summed values ​​is calculated to generate the second image residual, thereby ensuring the accuracy of the generated second image residual. Then, the first preset threshold corresponding to the second image residual is obtained, and then, each element in the second image residual is compared with the first preset threshold, thereby ensuring the accuracy of the obtained comparison result. Then, according to the comparison result, the second image residual is changed into a binary matrix, thereby ensuring the accuracy of the generated binary matrix. The binary matrix and the first image residual are subjected to an AND operation to generate a target residual matrix, thereby ensuring the accuracy of the generated target residual matrix, thereby ensuring the accuracy of the generated additional information. Next, the encoder obtains the second preset threshold corresponding to the target residual matrix, and based on the relationship between the second preset threshold and each element in the target residual matrix, performs thresholding processing on the target residual matrix to generate the first candidate additional information, thereby ensuring the accuracy of the generated first candidate additional information. Then, the first candidate additional information is subjected to DCT conversion to generate the second candidate additional information, thereby ensuring the accuracy of the generated second candidate additional information. The second candidate additional information is scanned and encoded to generate additional information, thereby ensuring the accuracy of the generated additional information.

[0157] , thereby ensuring that the identification information in the decoded video frame is clear.

[0158] In one embodiment of the present application, Figure 5 As shown, a video encoding method is provided, which is described by taking the method applied to an encoding end as an example, and includes the following steps:

[0159] S31. Monitor input events.

[0160] For more information about this step, see Figure 2 The introduction of S21 will not be elaborated here.

[0161] S32: When an input event is monitored, the current video data frame is encoded to generate encoded data.

[0162] For more information about this step, see Figure 2 The introduction of S22 will not be elaborated here.

[0163] S33, reconstructing the encoded data to generate a reconstructed video data frame.

[0164] For more information about this step, see Figure 2 The introduction of S23 will not be elaborated here.

[0165] S34: Generate additional information corresponding to the current video data frame based on the difference between the current video data frame and the reconstructed video data frame.

[0166] The additional information is used to represent identification information that needs to be displayed in the current video data frame.

[0167] For more information about this step, see Figure 2 The introduction of S24 will not be elaborated here.

[0168] S35, transmitting the encoded data and the additional information to the decoding end corresponding to the encoding end.

[0169] In an optional implementation manner of the present application, the above S35 “transmitting the encoded data and the additional information to a decoding end corresponding to the encoding end” may include the following steps:

[0170] S351. Analyze the additional information and generate identification information corresponding to the additional information.

[0171] Specifically, after generating the additional information, the encoder may use identification information corresponding to the additional information.

[0172] For example, the encoder can set a coding flag in the coding format. If there is additional information, the flag is recorded as 1. If no input event is detected, no additional information needs to be generated and the flag is recorded as 0. This indicates to the decoder whether there is additional information in the current video data frame decoding.

[0173] S352: Transmit the encoded data, the additional information, and the identification information corresponding to the additional information to a decoding end corresponding to the encoding end.

[0174] Specifically, after generating the additional information and the identification information corresponding to the additional information, the encoding end may transmit the encoded data, the additional information and the identification information corresponding to the additional information to the decoding end corresponding to the encoding end based on the communication connection with the decoding end.

[0175] The video encoding method provided by the embodiment of the present invention analyzes the additional information and generates identification information corresponding to the additional information; the encoded data, the additional information and the identification information corresponding to the additional information are transmitted to the decoding end corresponding to the encoding end, thereby ensuring that the decoding end can accurately determine whether there is an additional message based on the identification information corresponding to the additional information, and there is no need to decode and read the additional information before determining whether there is additional information, thereby reducing the workload of the decoding end and improving the working efficiency of the decoding end.

[0176] It should be noted that the image decoding method provided in the embodiment of the present application may be executed by an image decoding device, which may be implemented as part or all of a decoding end device by software, hardware, or a combination of software and hardware, wherein the decoding end may be a server or a terminal, wherein the server in the embodiment of the present application may be a single server or a server cluster composed of multiple servers, and the terminal in the embodiment of the present application may be a smart phone, a personal computer, a tablet computer, a wearable device, an intelligent robot, or other intelligent hardware devices. In the following method embodiments, the execution subject is taken as an example for explanation.

[0177] In one embodiment of the present application, Figure 6 As shown, a video decoding method is provided, which is described by taking the method applied to a decoding end as an example, and includes the following steps:

[0178] S41. Receive the encoded data and additional information sent by the encoding end.

[0179] The encoded data and the additional information are generated according to the video encoding method of any one of the above-mentioned implementation modes.

[0180] Specifically, the decoding end may receive the encoded data and additional information sent by the encoding end based on the communication connection with the encoding end.

[0181] S42: Decode and reconstruct the encoded data to generate a reconstructed video data frame.

[0182] Specifically, after receiving the encoded data and the additional information, the decoding end may decode and reconstruct the encoded data to generate a reconstructed video data frame.

[0183] S43, performing inverse DCT transformation on the additional information to generate a target additional matrix.

[0184] Specifically, the decoding end performs an inverse DCT transform on the additional information to generate a target additional matrix.

[0185] S44: Based on the relationship between the target additional matrix and the reconstructed video data frame, the reconstructed video data frame is optimized to generate a target reconstructed video frame.

[0186] In an optional implementation manner of the present application, the above S44 “optimizing the reconstructed video data frame based on the relationship between the target additional matrix and the reconstructed video data frame to generate a target reconstructed video frame” may include the following steps:

[0187] S441, performing horizontal difference processing and vertical difference processing on the reconstructed video data frame, and generating candidate image residuals based on the relationship between the generated matrices.

[0188] Specifically, the reconstructed video data frame is subjected to horizontal differential residual processing to obtain a reconstructed horizontal differential matrix; the reconstructed video data frame is subjected to vertical differential residual processing to obtain a reconstructed vertical differential matrix, and then the square root calculation is performed on the corresponding elements in the reconstructed horizontal differential matrix and the reconstructed vertical differential matrix to generate a candidate image residual.

[0189] Specifically, the encoding end may perform square calculations on corresponding elements in the reconstructed horizontal difference matrix and the reconstructed vertical difference matrix, sum the squared data, and perform square root calculations on the summed values ​​to generate a second image residual.

[0190] Exemplarily, assuming that the reconstructed horizontal difference matrix and the reconstructed vertical difference matrix are both 2×2 matrices, the encoding can perform square calculations on the elements in the first row and the first column of the reconstructed horizontal difference matrix and the reconstructed vertical difference matrix respectively, and then sum the data obtained after the square calculations, and then perform square root calculations on the summed data to generate the elements in the first row and the first column of the candidate image residual, and then calculate the values ​​of other elements in the candidate image residual in sequence to generate the candidate image residual.

[0191] S442: construct a target optimization function based on the difference between the candidate image residual and the target additional matrix.

[0192] Specifically, the decoding end may construct a target optimization function based on the difference between the candidate image residual and the target additional matrix.

[0193] Exemplarily, the objective optimization function may be as shown in the following formula (1):

[0194]

[0195] Where Y represents the reconstructed video data frame without additional information, X represents the target additional matrix, D is the root mean square operator after horizontal and vertical differences, and DY represents the candidate image residual.

[0196] The original intention of the above formula is that the difference of the reconstructed video data frame should be close enough to the edge difference of the current video data, which is used as a condition to constrain the edge of the reconstructed video data frame, so as to better restore the edge of the identification image of the reconstructed video data frame.

[0197] S443, solving the target optimization function, and updating the reconstructed video data frame according to the solution result to generate a target reconstructed video frame.

[0198] Specifically, the decoding end solves the target optimization function, updates the reconstructed video data frame according to the solution result, and generates the target reconstructed video frame.

[0199] Exemplarily, the decoding end may adopt the constraint condition of L1 normal, the entire constraint term is non-convex, and an auxiliary variable D' is introduced in the solution process, as shown in the following formula (2):

[0200]

[0201] The irreversible property of the differential operator D is resolved, and the final target reconstructed video frame is solved by using the ADMM algorithm through step-by-step iterative convergence.

[0202] S45, superimposing the target reconstructed video frame and the target additional matrix, rendering and then outputting for display.

[0203] Specifically, after the decoding end generates the target reconstructed video frame, the decoding end may superimpose the target reconstructed video frame with the target additional matrix and render the resultant to output and display.

[0204] For example, Figure 7 As shown, a schematic diagram of decoding the coded data and the additional information at the decoding end is shown. Specifically, the coded data is decoded and reconstructed to generate a reconstructed video data frame; the additional information is subjected to an inverse DCT transform to generate a target additional matrix; based on the relationship between the target additional matrix and the reconstructed video data frame, the reconstructed video data frame is optimized to generate a target reconstructed video frame. The target reconstructed video frame is superimposed on the target additional matrix and rendered for output display.

[0205] It should be noted that the superimposed target reconstructed video frame cannot be added to the decoding reference frame queue at this time, which will break the consistency with the encoding reference frame queue. Therefore, a copy needs to be added to the decoding queue after the first decoding and reconstruction.

[0206] The video decoding method provided by the embodiment of the present invention receives the coded data and additional information sent by the coding end, then decodes and reconstructs the coded data to generate a reconstructed video data frame, thereby ensuring the accuracy of the generated reconstructed video data frame. Then, the additional information is subjected to an inverse DCT transform to generate a target additional matrix, thereby ensuring the accuracy of the generated target additional matrix. The reconstructed video data frame is subjected to horizontal differential processing and vertical differential processing, and based on the relationship between the generated matrices, a candidate image residual is generated, thereby ensuring the accuracy of the generated candidate image residual. Based on the difference between the candidate image residual and the target additional matrix, a target optimization function is constructed, thereby ensuring the accuracy of the constructed target optimization function. Then, the target optimization function is solved, and according to the solution result, the reconstructed video data frame is updated to generate a target reconstructed video frame, thereby ensuring the accuracy of the generated target reconstructed video frame. The target reconstructed video frame is superimposed on the target additional matrix and rendered for output display, thereby ensuring the clarity of the output displayed target reconstructed video frame and the clarity of the identification information in the target reconstructed video frame.

[0207] In order to better explain the video encoding method and the video decoding method provided in the embodiment of the present application, the embodiment of the present application provides an overall flow chart of the video encoding method and the video decoding method, as shown in FIG. Figure 8 As shown. Specifically, the encoder monitors the input event, and when the input event is monitored, the current video data frame is encoded to generate encoded data, and the encoded data is reconstructed to generate a reconstructed video data frame. The reconstructed video data frame is subtracted from the current video data frame to obtain the first image residual. Then, the current video data frame is subjected to horizontal differential residual processing to obtain a horizontal differential matrix, and the current video data frame is subjected to vertical differential residual processing to obtain a vertical differential matrix. The corresponding elements in the horizontal differential matrix and the vertical differential matrix are squared, and the squared data are summed, and the square root of the summed value is calculated to generate the second image residual. The first preset threshold corresponding to the second image residual is obtained; each element in the second image residual is compared with the first preset threshold; according to the comparison result, the second image residual is changed into a binary matrix; and the binary matrix is ​​ANDed with the first image residual to generate a target residual matrix. Obtain a second preset threshold corresponding to the target residual matrix; perform thresholding processing on the target residual matrix based on the relationship between the second preset threshold and each element in the target residual matrix to generate first candidate additional information; perform DCT conversion on the first candidate additional information to generate second candidate additional information; perform scanning and encoding processing on the second candidate additional information to generate additional information. Analyze the additional information to generate identification information corresponding to the additional information; transmit the encoded data, the additional information, and the identification information corresponding to the additional information to the decoding end corresponding to the encoding end.

[0208] The decoding end receives the coded data and additional information sent by the encoding end, decodes and reconstructs the coded data to generate a reconstructed video data frame; performs inverse DCT transformation on the additional information to generate a target additional matrix. Performs horizontal and vertical differential processing on the reconstructed video data frame, and generates a candidate image residual based on the relationship between the generated matrices. Based on the difference between the candidate image residual and the target additional matrix, a target optimization function is constructed; the target optimization function is solved, and based on the solution result, the reconstructed video data frame is updated to generate a target reconstructed video frame. The target reconstructed video frame is superimposed on the target additional matrix and rendered for output display.

[0209] It should be understood that although Figure 1 , Figure 2 , Figure 5 ,as well as Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 , Figure 2 , Figure 5 ,as well as Figure 6 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0210] like Fig. 9 As shown, this embodiment provides an image encoding device, which is applied to an encoding end; comprising:

[0211] The monitoring module 51 is used to monitor input events.

[0212] The encoding process 52 is used to perform encoding process on the current video data frame to generate encoded data when an input event is monitored.

[0213] The first reconstruction module 53 is used to reconstruct the encoded data to generate a reconstructed video data frame.

[0214] The generating module 54 is used to generate additional information corresponding to the current video data frame based on the difference between the current video data frame and the reconstructed video data frame; the additional information is used to represent identification information that needs to be displayed in the current video data frame.

[0215] The transmission module 55 is used to transmit the encoded data and the additional information to the decoding end corresponding to the encoding end.

[0216] In one embodiment of the present application, the above-mentioned generation module 54 is specifically used to obtain a first image residual by subtracting a reconstructed video data frame from a current video data frame; performing horizontal differential residual processing on the current video data frame to obtain a horizontal differential matrix; performing vertical differential residual processing on the current video data frame to obtain a vertical differential matrix; generating a second image residual based on the relationship between the horizontal differential matrix and the vertical differential matrix; generating a target residual matrix based on the relationship between the first image residual and the second image residual; and analyzing the target residual matrix to generate additional information.

[0217] In one embodiment of the present application, the above-mentioned generation module 54 is specifically used to square the corresponding elements in the horizontal difference matrix and the vertical difference matrix, sum the squared data, and perform square root calculation on the summed value to generate a second image residual.

[0218] In one embodiment of the present application, the above-mentioned generation module 54 is specifically used to obtain a first preset threshold corresponding to the second image residual; compare each element in the second image residual with the first preset threshold; according to the comparison result, change the second image residual into a binary matrix; perform an AND operation on the binary matrix and the first image residual to generate a target residual matrix.

[0219] In one embodiment of the present application, the above-mentioned generation module 54 is specifically used to obtain a second preset threshold corresponding to the target residual matrix; based on the relationship between the second preset threshold and each element in the target residual matrix, perform thresholding processing on the target residual matrix to generate first candidate additional information; perform DCT conversion on the first candidate additional information to generate second candidate additional information; scan and encode the second candidate additional information to generate additional information.

[0220] In one embodiment of the present application, the transmission module 55 is specifically used to analyze the additional information, generate identification information corresponding to the additional information, and transmit the encoded data, the additional information, and the identification information corresponding to the additional information to the decoding end corresponding to the encoding end.

[0221] like Fig.10 As shown, this embodiment provides an image decoding device, which is applied to a decoding end; comprising:

[0222] The receiving module 61 is used to receive the encoded data and additional information sent by the encoding end, where the encoded data and additional information are generated according to the video encoding method of the first aspect and any one of the implementation methods of the first aspect.

[0223] The second reconstruction module 62 is used to decode and reconstruct the encoded data to generate a reconstructed video data frame.

[0224] An inverse transform module 63 is used to perform an inverse DCT transform on the additional information to generate a target additional matrix;

[0225] The optimization module 64 is used to optimize the reconstructed video data frame based on the relationship between the target additional matrix and the reconstructed video data frame to generate a target reconstructed video frame.

[0226] The superposition module 65 is used to superimpose the target reconstructed video frame and the target additional matrix and render them for output display.

[0227] In one embodiment of the present application, the above-mentioned optimization module 64 is specifically used to perform horizontal difference processing and vertical difference processing on the reconstructed video data frame, and generate a candidate image residual based on the relationship between the generated matrices; construct a target optimization function based on the difference between the candidate image residual and the target additional matrix; solve the target optimization function, and update the reconstructed video data frame according to the solution result to generate a target reconstructed video frame.

[0228] For the specific limitations and beneficial effects of the image encoding device and the image decoding device, please refer to the limitations of the video encoding method and the video decoding method above, which will not be repeated here. The various modules in the above-mentioned image encoding device and the image decoding device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the encoding end or / and the decoding end in hardware form, or can be stored in the memory in the encoding end or / and the decoding end in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0229] The embodiment of the present invention also provides an encoding end having the above Fig. 9 The image encoding device shown in the figure also provides a decoding end having the above Fig.10 The image decoding device shown.

[0230] like Fig.11 As shown, Figure 1 is a schematic diagram of the structure of an encoding end and / or a decoding end provided by an optional embodiment of the present invention, such as Fig.11As shown, the encoding end and / or decoding end may include: at least one processor 71, such as a CPU (Central Processing Unit), at least one communication interface 73, a memory 74, and at least one communication bus 72. The communication bus 72 is used to realize the connection and communication between these components. The communication interface 73 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 73 may also include a standard wired interface and a wireless interface. The memory 74 may be a high-speed RAM memory (Random Access Memory) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 74 may optionally be at least one storage device located away from the aforementioned processor 71. The processor 71 may be combined with Fig. 9 or / and Fig.10 In the described device, the memory 74 stores an application program, and the processor 71 calls the program code stored in the memory 74 to execute any of the above method steps.

[0231] The communication bus 72 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 72 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0232] Among them, the memory 74 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 74 may also include a combination of the above types of memory.

[0233] The processor 71 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.

[0234] The processor 71 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0235] Optionally, the memory 74 is also used to store program instructions. The processor 71 can call the program instructions to implement the present application Figure 1 , Figure 2 as well as Figure 5 The video encoding method shown in the embodiment or / and Figure 6 The image decoding shown in the embodiment.

[0236] The embodiment of the present invention further provides a non-transitory computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the video encoding method and the video decoding method in any of the above method embodiments. The storage medium may be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above types of memories.

[0237] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A video encoding method, characterized in that: Applied to the encoding end; including: Listen for input events; When the input event is monitored, encoding the current video data frame to generate encoded data; Reconstructing the encoded data to generate a reconstructed video data frame; Based on the difference between the current video data frame and the reconstructed video data frame, additional information corresponding to the current video data frame is generated; the additional information is used to represent identification information that needs to be displayed in the current video data frame; Transmitting the encoded data and the additional information to a decoding end corresponding to the encoding end; The step of generating the additional information corresponding to the current video data frame based on the difference between the current video data frame and the reconstructed video data frame includes: Subtracting the reconstructed video data frame from the current video data frame to obtain a first image residual; Performing horizontal difference residual processing on the current video data frame to obtain a horizontal difference matrix; Performing longitudinal difference residual processing on the current video data frame to obtain a longitudinal difference matrix; generating a second image residual based on the relationship between the transverse difference matrix and the longitudinal difference matrix; generating a target residual matrix based on a relationship between the first image residual and the second image residual; The target residual matrix is ​​analyzed to generate the additional information.

2. The method according to claim 1, characterized in that The generating a second image residual based on the relationship between the horizontal difference matrix and the vertical difference matrix includes: The corresponding elements in the horizontal difference matrix and the vertical difference matrix are squared, the squared data are summed, and the square root of the summed value is calculated to generate the second image residual.

3. The method according to claim 1, characterized in that: The generating a target residual matrix based on the relationship between the first image residual and the second image residual includes: Obtaining a first preset threshold corresponding to the second image residual; Comparing each element in the second image residual with the first preset threshold; According to the comparison result, the second image residual is changed into a binary matrix; An AND operation is performed on the binary matrix and the first image residual to generate the target residual matrix.

4. The method according to claim 1, characterized in that The step of analyzing the target residual matrix to generate the additional information includes: Obtaining a second preset threshold corresponding to the target residual matrix; Based on the relationship between the second preset threshold and each element in the target residual matrix, performing threshold processing on the target residual matrix to generate first candidate additional information; Performing DCT transformation on the first candidate additional information to generate second candidate additional information; The second candidate additional information is scanned and encoded to generate the additional information.

5. The method according to claim 1, characterized in that The transmitting the encoded data and the additional information to a decoding end corresponding to the encoding end includes: Analyze the additional information to generate identification information corresponding to the additional information; The encoded data, the additional information, and identification information corresponding to the additional information are transmitted to a decoding end corresponding to the encoding end.

6. A video decoding method, characterized in that: Applied to the decoding end; including: Receiving coded data and additional information sent by a coding end, wherein the coded data and the additional information are generated by the video coding method according to any one of claims 1 to 5; Decoding and reconstructing the encoded data to generate a reconstructed video data frame; Performing an inverse DCT transform on the additional information to generate a target additional matrix; Based on the relationship between the target additional matrix and the reconstructed video data frame, the reconstructed video data frame is optimized to generate a target reconstructed video frame; The target reconstructed video frame and the target additional matrix are superimposed and rendered for output display.

7. The method according to claim 6, characterized in that The reconstructed video data frame is optimized based on the relationship between the target additional matrix and the reconstructed video data frame to generate a target reconstructed video frame, Performing horizontal difference processing and vertical difference processing on the reconstructed video data frame, and generating candidate image residuals based on the relationship between the generated matrices; constructing a target optimization function based on the difference between the candidate image residual and the target additional matrix; The target optimization function is solved, and according to the solution result, the reconstructed video data frame is updated to generate the target reconstructed video frame.

8. An encoding end and / or a decoding end, characterized in that: It comprises a memory and a processor, wherein the memory stores computer instructions, and the processor executes the video encoding method according to any one of claims 1 to 5 and the video decoding method according to any one of claims 6 to 7 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the video encoding method described in any one of claims 1 to 5 and the video decoding method described in any one of claims 6 to 7.

Images