Video encoding, decoding method and related equipment

Abstract

The embodiment of the present application discloses a video encoding and decoding method and related equipment. Among them, the encoding device processes the sign language animation by: finding the identifier of the sign language animation corresponding to the voice information in the sign language library, and then encoding the identifier of the sign language animation. The decoding device decodes the encoded information to obtain the identifier of the sign language animation, finds the sign language animation corresponding to the identifier of the sign language animation in the sign language library, and fuses each frame of the animation image in the sign language animation with each frame of the decoded video to obtain a video embedded with the sign language animation. In this way, it is possible to reduce the data flow occupied by the encoding and decoding of the sign language animation, and improve the efficiency of the encoding device and the decoding device.

Description

Technical Field

[0001] The present application relates to the field of video communication technology, and in particular to a video encoding and decoding method and related equipment. Background Art

[0002] In order to enable hearing-impaired users to watch videos normally, a sign language video corresponding to the voice information in the video can be inserted into the video. This requires recording a sign language video sequence for the voice information in the video, inserting the sign language video sequence into the video sequence, and then encoding the synthesized video sequence. However, in the process of encoding the synthesized video sequence, the encoding of the sign language video sequence will occupy more data streams. Summary of the invention

[0003] The embodiment of the present application discloses a video encoding and decoding method and related equipment. Among them, the encoding device finds the identifier of the sign language animation corresponding to the voice information in the sign language library, and then encodes the identifier of the sign language animation. The decoding device obtains the identifier of the sign language animation according to the encoding information, searches the sign language animation corresponding to the identifier of the sign language animation in the sign language library, and fuses each frame of the animation image in the sign language animation with each frame of the image in the video to be processed to obtain a video embedded with the sign language animation. In this way, it is possible to reduce the data flow occupied by the encoding and decoding of the sign language animation.

[0004] In a first aspect, an embodiment of the present application provides a video encoding method, which is executed by an encoding device, and the method includes: obtaining text information of first voice information and first video information, the first voice information and the first video information are obtained in the same time period, the text information of the first voice information includes a sentence to be encoded, and the sentence to be encoded consists of M words, where M is a positive integer; searching for sign language animations corresponding to the M words in a sign language library to obtain identifiers of the M sign language animations, the M sign language animations correspond one-to-one to the M words, and the sign language library includes multiple words and identifiers of sign language animations corresponding to the multiple words; encoding the identifiers of the M sign language animations to obtain a first data stream; encoding the first video information to obtain a second data stream; and transmitting the first data stream and the second data stream to a decoding device.

[0005] In the above method, the encoding device searches the sign language library for the identifiers of the sign language animations corresponding to the words included in the sentence to be encoded, and encodes the identifiers of the sign language animations found, without encoding the sign language animations. The amount of data occupied by the encoding information of the identifiers of the sign language animations is less than the amount of data occupied by the encoding information of the sign language animations, so that less computing resources and storage resources can be used on the encoding device side to process information such as the identifiers of the sign language animations, thereby improving the encoding efficiency of the encoding device.

[0006] In combination with the first aspect, in some embodiments, the identifiers of the M sign language animations are arranged in the order of the words in the sentence to be encoded.

[0007] In combination with the first aspect, in some embodiments, the encoding of the identifiers of the M sign language animations to obtain a first data stream includes: for each of the identifiers of the M sign language animations, determining that the encoding information of each identifier is the difference between each identifier and the previous identifier of each identifier; the first data stream includes the encoding information of each identifier.

[0008] In combination with the first aspect, in some embodiments, the method further includes: obtaining an identifier of the first video information; encoding the identifier of the first video information to obtain encoding information of the identifier of the first video information; and the first data stream also includes encoding information of the identifier of the first video information.

[0009] In combination with the first aspect, in some embodiments, the method also includes: obtaining indication information, wherein the indication information is used to indicate the total number of image frames of the first video information; encoding the indication information to obtain encoding information of the indication information; and the first data stream also includes the encoding information of the indication information.

[0010] In combination with the first aspect, in some embodiments, the indication information includes the duration of the first voice information and the encoding rate of the first video information, and the encoding of the indication information to obtain the encoding information of the indication information includes: determining the encoding information of the duration of the first voice information as a first difference, wherein the first difference is the difference between the duration of the first voice information and the duration of a previous voice information of the first voice information; the first data stream also includes the encoding information of the duration of the first voice information; determining the encoding information of the encoding rate of the first video information as a second difference, wherein the second difference is the difference between the encoding rate of the first video information and the encoding rate of a previous video information of the first video information; the first data stream also includes the encoding information of the encoding rate of the first video information.

[0011] In combination with the first aspect, in some embodiments, the method also includes: receiving second voice information and second video information; converting the second voice information into text to obtain text information of the second voice information; identifying sentences in the text information of the second voice information to obtain at least one sentence; determining the sentence to be encoded from the at least one sentence, wherein the sentence to be encoded is a sentence; determining that the first voice information is the voice information corresponding to the sentence to be encoded in the second voice information; and determining that the first video information is the video information corresponding to the sentence to be encoded in the second video information.

[0012] In a second aspect, an embodiment of the present application provides a video decoding method, characterized in that it is applied to a decoding device, the method comprising: receiving a first data stream and a second data stream, the first data stream being obtained by encoding the identifiers of M sign language animations, the identifiers of the M sign language animations comprising identifiers of the sign language animations corresponding to M words respectively, the text information of the first voice information comprising a sentence to be decoded, the sentence to be decoded being composed of the M words, the second data stream being obtained by encoding the first video information, the first voice information and the first video information being acquired in the same time period; decoding the first data stream to obtain the identifiers of the M sign language animations; searching for the sign language animations corresponding to the identifiers of the M sign language animations in a sign language library to obtain M sign language animations, the sign language library comprising multiple sign language animations and identifiers corresponding to the multiple sign language animations respectively; decoding the second data stream to obtain the first video information; inserting the M sign language animations into the first video information to obtain the second video information.

[0013] In the above method, the decoding device decodes the decoding information of the sign language animation identifier to obtain the sign language animation identifier, searches the sign language animation corresponding to the sign language animation identifier in the sign language library, and then inserts the searched sign language animation into the decoded first video information. In this way, the decoding device can use less computing resources to obtain the sign language animation, thereby improving the decoding efficiency of the encoding device.

[0014] In combination with the second aspect, in some embodiments, the identifiers of the M sign language animations are arranged in order of the words in the sentence to be decoded, the first data stream includes encoding information of each of the M identifiers of the sign language animations, the encoding information of each identifier is the difference between each identifier and the previous identifier of each identifier, and decoding the first data stream to obtain the identifiers of the M sign language animations includes: for the encoding information of each identifier, determining that the decoding information of each identifier is the sum of the difference and the previous identifier of each identifier.

[0015] In combination with the second aspect, in some embodiments, the first data stream also includes encoded information of an identifier of the first video information, and decoding the first data stream further includes: decoding the encoded information of the identifier of the first video information to obtain the identifier of the first video information.

[0016] In combination with the second aspect, in some embodiments, the first data stream also includes the encoding information of the indication information, and the decoding of the first data stream also includes: decoding the encoding information of the indication information to obtain the indication information, and the indication information is used to indicate the total number of image frames of the first video information.

[0017] In combination with the second aspect, in some embodiments, the encoding information of the indication information includes encoding information of the duration of the first voice information and encoding information of the encoding rate of the first video information, the encoding information of the duration of the first voice information is a first difference, the first difference is the difference between the duration of the first voice information and the duration of a previous voice information of the voice information, the encoding information of the encoding rate of the first video information is a second difference, the second difference is the difference between the encoding rate of the first video information and the encoding rate of a previous video information of the first video information, and the decoding of the encoding information of the indication information to obtain the indication information includes: decoding the encoding information of the duration of the first voice information to obtain the duration of the first voice information, the duration of the first voice information being the sum of the first difference and the duration of the previous voice information; decoding the encoding information of the encoding rate of the first video information to obtain the encoding rate of the first video information, the encoding rate of the first video information being the sum of the second difference and the encoding rate of the previous video information of the first video information.

[0018] In combination with the second aspect, in some embodiments, inserting the M sign language animations into the first video information to obtain the second video information includes: selecting animation images with a target number of frames from the M sign language animations to obtain the sign language animation to be inserted, the target number of frames being no greater than the total number of image frames of the first video information; fusing the i-th frame animation image of the M sign language animations with the i-th frame image of the first video information to obtain the i-th frame image of the second video information, where i is a positive integer no greater than the total number of image frames of the M sign language animations.

[0019] In combination with the second aspect, in some embodiments, selecting an animation image with a target number of frames from the M sign language animations to obtain the sign language animation to be inserted includes: selecting one frame of animation image from the M sign language animations every preset frame to obtain the sign language animation to be inserted, the preset frame is a positive integer not less than the quotient of the total number of image frames of the M sign language animations and the total number of image frames of the first video information, and the target number of frames is the total number of image frames of the sign language animation to be inserted.

[0020] In a third aspect, an embodiment of the present application provides a coding device, comprising: a memory and a processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a fourth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the method described in the second aspect and any possible implementation method of the second aspect.

[0022] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the encoding device implements the video encoding method in the above-mentioned first aspect and its various possible implementations, or the decoding device implements the video decoding method in the above-mentioned second aspect and its various possible implementations.

[0023] It can be understood that the device provided in the third aspect is used to execute the method provided in the first aspect, the device provided in the fourth aspect is used to execute the method provided in the second aspect, and the computer-readable storage medium provided in the fifth aspect is used to execute the methods provided in the first and second aspects. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram of a video encoding and decoding system provided in an embodiment of the present application;

[0025] Figure 2 It is a flowchart of a video encoding method provided by an embodiment of the present application;

[0026] Figure 3 It is a flowchart of a method for obtaining first voice information and first video information provided by an embodiment of the present application;

[0027] Figure 4A is a schematic diagram of a first data stream and a second data stream provided in an embodiment of the present application;

[0028] Figure 4B It is a flowchart of a sign language animation identification encoding process provided by an embodiment of the present application;

[0029] Figure 5 It is a flowchart of a video decoding method provided by an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of the structure of an encoding device provided in an embodiment of the present application;

[0031] Figure 7 is a structural diagram of a decoding device provided in an embodiment of the present application;

[0032] Figure 8is a structural schematic diagram of an encoding device provided in an embodiment of the present application;

[0033] Fig. 9 It is a structural diagram of a decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The following embodiments of the present application provide a video encoding and decoding method and related devices.

[0037] The following introduces a coding and decoding system involved in an embodiment of the present application.

[0038] Figure 1 A schematic diagram of a coding and decoding system is shown in FIG. Figure 1 As shown, the system includes an encoding device 11 and a decoding device 12.

[0039] The encoding device 11 and the decoding device 12 may include a sign language library, or the sign language library is stored in other devices, such as a database, and the encoding device 11 and the decoding device 12 may obtain the sign language library or the data in the sign language library from the database. The sign language library may include multiple words, sign language animations corresponding to the multiple words, and identifiers of each sign language animation. The identifier of the sign language animation may be a number, and each sign language animation has a unique number corresponding to it.

[0040] The encoding device 11 is a device for compressing information. The encoding device 11 obtains video information and voice information through a camera, a microphone and other devices, finds the identifier of the sign language animation corresponding to the voice information in the sign language library according to the voice information, and then encodes the identifier of the sign language animation and the video to be processed to obtain the first data stream 13 and the second data stream 14. The encoding process may be that the encoding device 11 removes redundant information such as time redundancy and space redundancy between information, only encodes abnormal components in the information, and then compresses the above information into the first data stream 13 and the second data stream 14 with smaller data volume. Figure 1 As shown in FIG. 1 , the first data stream 13 and the second data stream 14 are transmitted to the decoding device 12 , and the decoding device 12 decompresses the data stream 13 to obtain the original information.

[0041] The decoding device 12 is a device for decompressing data streams. The decoding device 12 receives the first data stream 13 and the second data stream 14, and decodes the first data stream 13 and the second data stream 14 through the inverse process of compression to obtain the identification of the sign language animation and the video to be processed. The decoding device 12 searches the sign language library for the sign language animation corresponding to the identification of the sign language animation, and then determines the number of animation image frames for the found sign language animation, and selects animation images from the found sign language animation according to the above number of frames to obtain the sign language animation to be inserted. Finally, the decoding device 12 merges each frame of the animation image in the sign language animation to be inserted with each frame of the image in the above video to be processed to obtain a video with the embedded sign language animation, and then the video with the embedded sign language animation can be sent to a display or other device.

[0042] It should be understood that the encoding device 11 and the decoding device 12 may use the sign language library as described above.

[0043] In some embodiments, the sign language libraries that can be used by the encoding device 11 and the decoding device 12 may not be exactly the same, that is, the sign language library used by the encoding device 11 includes multiple words, sign language animations corresponding to the multiple words, and identifiers of each sign language animation, while the sign language library used by the decoding device 12 includes multiple sign language animations and identifiers of each sign language animation but does not include multiple words, wherein in the sign language library used by the encoding device 11 and the sign language library used by the decoding device 12, the same identifier corresponds to the same sign language animation, and each sign language animation has a unique identifier corresponding to it.

[0044] See also Figure 2 , Figure 2 is a flow chart of a video encoding method, which can be performed by the above Figure 1 The encoding device 11 in the embodiment is executed, and it should be understood that Figure 2 As shown, the method may include but is not limited to the following steps:

[0045] S101. An encoding device obtains text information of first voice information and first video information.

[0046] The first voice information and the first video information are acquired in the same time period. For example, in an audio or video, the voice information with the text "Today is a good weather" starts at time a and ends at time b. The voice information and video information acquired in the time period from a to b are the first voice information and the first video information, respectively. The text information of the first voice information includes a sentence to be encoded, and the sentence to be encoded consists of M words, where M is a positive integer.

[0047] The text information of the first voice information is text converted by the encoding device using voice recognition technology from the first voice information.

[0048] The sentence to be encoded is at least one sentence included in the text information of the first voice information, consisting of M words, and the order of the M words is the time sequence of the voice recognition text. In some embodiments, the sentence to be encoded is one sentence, and the one sentence consists of M words. In other embodiments, the sentence to be encoded is a plurality of sentences, and the plurality of sentences consists of M words.

[0049] In the embodiment of the present application, the sentence to be encoded is taken as an example. Figure 2 It should be understood that the sentence to be encoded may also be multiple sentences, and the encoding device may still execute Figure 2 The video encoding method shown.

[0050] S101 may include but is not limited to the following steps:

[0051] S1011. The encoding device receives second voice information and second video information.

[0052] In the same time period, the encoding device collects the second voice information through the microphone and collects the second video information through the camera.

[0053] S1012. The encoding device converts the second voice information into text to obtain text information of the second voice information.

[0054] S1013. The encoding device recognizes sentences in the text information of the second voice information to obtain at least one sentence.

[0055] The at least one sentence is all the sentences in the text information of the second voice information. The at least one sentence changes in real time as the second voice information and the second video information are collected. The encoding device arranges the at least one sentence in the order of the time when the text is recognized from the second voice information.

[0056] S1014. The encoding device determines a sentence to be encoded from at least one sentence, where the sentence to be encoded is one sentence.

[0057] Specifically, the encoding device determines that the sentence to be encoded is the next sentence of the encoded sentence according to the order of at least one sentence. The encoding device then splits the sentence to be encoded into M words.

[0058] S1015. The encoding device determines that the first voice information is the voice information corresponding to the sentence to be encoded in the second voice information.

[0059] The voice information corresponding to the sentence to be encoded is the voice information of the sentence to be encoded whose text information is the sentence to be encoded.

[0060] S1016. The encoding device determines that the first video information is the video information corresponding to the sentence to be encoded in the second video information.

[0061] The encoding device determines the duration of the first voice information, which is the time period from the start time point to the end time point of the first voice information. The encoding device determines the first video information as the video information within the second video information from the above start time point to the end time point. The start time point and the end time point of the first video information are the same as the start time point and the end time point of the first voice information.

[0062] It should be understood that all images in the first video information are images to be encoded that are determined according to the encoding rate of the encoding device and the above-mentioned duration.

[0063] S102: The encoding device searches for sign language animations corresponding to M words in a sign language library to obtain identifiers of the M sign language animations.

[0064] Specifically, the encoding device searches for sign language animations corresponding to M words in the sign language library, and arranges the identifiers of the M sign language animations found according to the order of the words in the sentence to be encoded. The M sign language animations correspond to the M words one by one, and the sign language library includes multiple words and the identifiers of the sign language animations corresponding to the multiple words.

[0065] S103: The encoding device encodes the identifiers of the M sign language animations to obtain a first data stream.

[0066] like Figure 4A As shown, the first data stream may be a sequence of one or more coding blocks, and the coding block may include field 1, field 2, and field 3. Field 1 is used to store the identifier of the first video information or the encoding information of the identifier; field 2 is used to store the indication information or the encoding information of the indication information, and the indication information is used to indicate the total number of frames of the first video information; field 3 is used to store the encoding information of the identifier of one or more sign language animations, Figure 4AThe encoding information of the identification of two sign language animations is used as an example to illustrate. It should be understood that the above Figure 4A Taking the above three fields as an example, the coding block can also include fewer fields or more fields, which is not limited here.

[0067] In an implementation of S103, the encoding device may encode one or more of the identifiers of the M sign language animations, the identifier of the first video information, the indication information, etc. The encoding methods are described below respectively:

[0068] (1) Encoding of the identifiers of M sign language animations:

[0069] The encoding device determines, for each of the M sign language animation identifiers, that encoding information of each identifier is the difference between each identifier and the previous identifier of each identifier; the first data stream includes encoding information of each identifier.

[0070] For example, the sentence to be encoded is "Today is a good weather", which includes five words: "today", "yes", "good", and "weather". In the sign language library, the sign language animations corresponding to these five words are marked as: "x1", "x2", "x3", "x4", and "x5". The encoding process diagram is as follows: Figure 4B As shown, first determine the encoding information of the number of words included in the sentence to be encoded, then determine the encoding information of the first identifier x1, and finally determine the encoding information of each identifier in turn as the difference between each identifier and the previous identifier.

[0071] (2) Encoding of the identifier of the first video information:

[0072] In some embodiments, the identifier of the first video information may be a video number, which is used to indicate the video information corresponding to the identifiers of the M sign language animations, so that at the decoding end, the video information to be inserted into the M sign language animations may be determined based on the number. The encoding device determines that the encoding information of the identifier of the first video information is the difference between the number of the first video information and the number of the previous video information of the first video information.

[0073] It should be understood that the identifier of the first video information is used to distinguish two adjacent video information, and it can also be "1" or "0", in which case, no encoding is required.

[0074] (3) Encoding of indication information:

[0075] In some embodiments, the indication information includes the encoding rate of the encoding device for the first video information and the duration of the first voice information. The indication information may indicate the total number of image frames of the first video information, which is the product of the encoding rate and the duration of the first voice information.

[0076] The encoding device determines that the encoding information of the duration of the first voice information is a first difference value, where the first difference value is a difference between the duration of the first voice information and the duration of a previous voice information of the first voice information.

[0077] The encoding device determines that the encoding information of the encoding rate of the first video information is the second difference, the second difference is the difference between the encoding rate of the first video information and the encoding rate of the previous video information of the first video information, and the previous video information of the first video information is the video information corresponding to the previous voice information of the first voice information.

[0078] In some embodiments, the indication information is the total number of image frames of the first video information. The encoding device determines that the total number of image frames of the first video information is the third difference, and the third difference is the difference between the total number of image frames of the first video information and the total number of image frames of the previous video information of the first video information, and the previous video information of the first video information is the video information corresponding to the previous voice information of the first voice information.

[0079] S104: The encoding device encodes the first video information to obtain a second data stream.

[0080] like Figure 4A As shown, the second data stream may be a sequence of one or more coding blocks, and the coding block may include field 4 and field 5. Field 4 is used to store the identifier of the first video information or the coding information of the identifier; field 5 is used to store the coding information of at least one image. For the identifier of the first video information and the coding method of the identifier, please refer to the relevant description in the above step S103, which will not be repeated here.

[0081] In one implementation, the encoding device removes redundant information from the first video information and only performs entropy encoding on the entropy in the first video information. The first video information includes entropy and redundant information, the entropy is the abnormal component between information in the first video information, and is non-repeated information in the first video information, the redundant information is the rest of the first video information, and is repeated information, and the redundant information includes spatial redundancy, temporal redundancy, visual redundancy, etc.

[0082] S105: The encoding device sends the first data stream and the second data stream to the decoding device.

[0083] It should be understood that the encoding device can perform the above steps S102-S103 and S104 simultaneously. In some embodiments, steps S102, S103, and S104 can also be performed sequentially, which is not limited in the embodiments of the present application.

[0084] See also Figure 5 , Figure 5 is a flow chart of a video decoding method, which can be performed by the above Figure 1The decoding device 12 in the embodiment is executed, and it should be understood that Figure 5 As shown, the method may include but is not limited to the following steps:

[0085] S201. A decoding device receives a first data stream and a second data stream, wherein the first data stream is obtained by encoding M sign language animation identifiers, the M sign language animation identifiers include identifiers of sign language animations corresponding to M words respectively, the text information of the first voice information includes a sentence to be decoded, the sentence to be decoded consists of M words, the second data stream is obtained by encoding the first video information, and the first voice information and the first video information are acquired in the same time period.

[0086] It should be understood that M can be different positive integers in different data streams.

[0087] The related description of the first data stream and the second data stream can refer to the above Figure 2 The description of the first data stream and the second data stream in steps S103 and S104 in the method embodiment shown are not repeated here.

[0088] S202: The decoding device decodes the first data stream to obtain identifiers of M sign language animations.

[0089] In an implementation of S202, the encoding device may decode one or more of the identifiers of the M sign language animations, the identifier of the first video information, the indication information, etc. The decoding methods are described below respectively:

[0090] (1) Decoding of M sign language animation logos:

[0091] During encoding, the encoding information of each identifier is the difference between each identifier and the previous identifier of each identifier. Therefore, at the decoding end, the decoding device determines the decoding information of each identifier as the sum of the difference and the previous identifier of each identifier for the encoding information of each identifier. The identifiers of the M sign language animations are arranged in the order of the words in the sentence to be decoded.

[0092] (2) Decoding of the identifier of the first video information.

[0093] In some embodiments, the identifier of the first video information may be a video number, which is used to indicate the video information corresponding to the identifiers of the M sign language animations, so that at the decoding end, the video information to be inserted into the M sign language animations can be determined based on the number. During encoding, the encoding device determines that the encoding information of the identifier of the first video information is the difference between the number of the first video information and the number of the previous video information of the first video information. Therefore, at the decoding end, the decoding device determines that the number of the first video information is the sum of the difference and the number of the previous video information of the first video information.

[0094] It should be understood that the identifier of the first video information is used to distinguish between two adjacent video information, and it can also be "1" or "0", in which case, decoding may not be performed.

[0095] (3) Decoding of indication information:

[0096] Specifically, the indication information is used to indicate the total number of image frames of the first video information.

[0097] In some embodiments, the coding information of the indication information includes coding information of the duration of the first voice information and coding information of the coding rate of the first video information. The coding device decodes the coding information of the duration of the first voice information to obtain the duration of the first voice information, which is the sum of the first difference and the duration of the previous voice information; the coding device decodes the coding information of the coding rate of the first video information to obtain the coding rate of the first video information, which is the sum of the second difference and the coding rate of the previous video information of the first video information. Among them, the coding information of the duration of the first voice information is the first difference, the first difference is the difference between the duration of the first voice information and the duration of the previous voice information of the voice information, the coding information of the coding rate of the first video information is the second difference, the second difference is the difference between the coding rate of the first video information and the coding rate of the previous video information of the first video information, and the previous video information of the first video information is the video information corresponding to the previous voice information of the first voice information. It should be understood that the product of the duration of the first voice information and the coding rate of the first video information is the total number of image frames of the first video information, and the duration of the first voice information is the same as the duration of the first video information.

[0098] In some other embodiments, the encoding information of the indication information includes encoding information of the total number of image frames of the first video information. The encoding device decodes the encoding information of the total number of image frames of the first video information to obtain the total number of image frames of the first video information, and the total number of image frames of the first video information is the sum of the third difference and the total number of image frames of the previous video information of the first video information. Among them, the encoding information of the total number of image frames of the first video information is the third difference, and the third difference is the difference between the total number of image frames of the first video information and the total number of image frames of the previous video information of the first video information, and the previous video information of the first video information is the video information corresponding to the previous voice information of the first voice information.

[0099] S203: The decoding device searches for the sign language animations corresponding to the identifiers of the M sign language animations in the sign language library to obtain the M sign language animations. The sign language library includes a plurality of sign language animations and the identifiers corresponding to the plurality of sign language animations.

[0100] It should be understood that the M sign language animations obtained by the decoding device are arranged in the order of the identifiers of the M sign language animations.

[0101] S204: The decoding device decodes the second data stream to obtain first video information.

[0102] In one implementation, the encoding device performs entropy decoding on the entropy in the first video information, and calculates the redundant information in the first video information to obtain the first video information. The first video information includes entropy and redundant information, the entropy is the abnormal component between the information in the first video information, and is the non-repeated information in the first video information, the redundant information is the rest of the first video information, and is the repeated information, and the redundant information includes spatial redundancy, temporal redundancy, visual redundancy, etc.

[0103] It should be understood that the first video information also includes an identifier of the first video information, and in some embodiments, the identifier can be used to indicate the first frame image of the first video information. Two adjacent video information can be identified based on the identifier.

[0104] S205. The decoding device inserts M sign language animations into the first video information to obtain second video information.

[0105] In one implementation of S205, S205 may include but is not limited to the following steps:

[0106] S2051, the decoding device determines whether the total number of frames of the M sign language animations is greater than the total number of image frames of the first video information. If so, the decoding device executes S2052: determining the target number of frames according to the total number of image frames of the first video information; otherwise, if the total number of frames of the M sign language animations is not greater than the total number of image frames of the first video information, the decoding device merges the i-th frame animation image of the M sign language animations with the i-th frame image of the first video information to obtain the i-th frame image of the second video information. Wherein, i is a positive integer not greater than the total number of image frames of the M sign language animations.

[0107] S2052: The decoding device determines a target frame number according to the total frame number of the first video information, wherein the target frame number is the total frame number of the image to be inserted into the sign language animation, and the target frame number is not greater than the total frame number of the first video information.

[0108] Specifically, the decoding device selects one frame of animation image from the M sign language animations every preset frame to obtain the sign language animation to be inserted, wherein the preset frame is a positive integer not less than the quotient of the total number of image frames of the M sign language animations and the total number of image frames of the first video information.

[0109] S2053. The decoding device fuses the i-th frame animation image to be inserted into the sign language animation with the i-th frame image in the first video information to obtain the i-th frame image of the second video information. Here, i is a positive integer not greater than the total number of frame images of the M sign language animations. The total number of frame images of the first video information is equal to the total number of frame images of the second video information.

[0110] When the total number of frame images in the sign language animation to be inserted is less than the total number of frame images of the first video information, for example, the sign language animation to be inserted includes P frame images, the first video information includes Q frame images, P < Q, and P and Q are positive integers. The first P frame images in the second video information are the fusion images of the first P frame images of the first video information and the P frame images included in the sign language animation to be inserted respectively, and the subsequent Q - P frame images in the second video information are the subsequent Q - P frame images of the first video information respectively.

[0111] It should be noted that in step S205, the decoding device can identify the first video information based on the identifier of the first video information. This identifier can indicate the first frame image of the first video information, that is, the decoding device starts inserting the sign language animation from this first frame image.

[0112] It should be understood that the decoding device executes the above steps S202 - S203 and S204 simultaneously. In some embodiments, the decoding device can also execute steps S202, S203, and S204 in sequence, which is not limited in the embodiments of the present application.

[0113] The method of the embodiments of the present application is elaborated in detail above, and the related apparatuses and devices of this embodiment are provided below.

[0114] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of an encoding device provided by an embodiment of the present application. The encoding device 600 can be the encoding device in the method embodiment shown above Figure 2 and the encoding device 600 can include a first acquisition unit 601, a second acquisition unit 602, a first encoding unit 603, a second encoding unit 604, and a sending unit 605. The detailed descriptions of each unit are as follows:

[0115] The first acquisition unit 601 can be used to acquire first speech information and first video information. Here, the first speech information and the first video information are acquired in the same time period, and the text information of the first speech information includes a sentence to be encoded, and the sentence to be encoded is composed of M words, where M is a positive integer;

[0116] The second acquisition unit 602 may be configured to acquire text information of the first voice information, and search for identifiers of sign language animations corresponding to the M words in a sign language library, wherein the M sign language animations correspond to the M words one-to-one, and the sign language library includes a plurality of words and identifiers of sign language animations corresponding to the plurality of words;

[0117] A first encoding unit 603 may be used to encode the identifiers of the M sign language animations to obtain a first data stream;

[0118] The second encoding unit 604 may be used to encode the first video information to obtain a second data stream;

[0119] The sending unit 605 may be configured to send the first data stream and the second data stream to a decoding device.

[0120] In some embodiments, the first data stream also includes encoding information of the identifier of the first video information and encoding information of the indication information, and the first encoding unit 603 is also used to encode the identifier and the indication information of the first video information, wherein the identifier of the first video information can be used to indicate the video information corresponding to the identifiers of M sign language animations, and the indication information is used to indicate the total number of image frames of the first video information.

[0121] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a decoding device provided in an embodiment of the present application. The decoding device 700 can be the above Figure 5 The decoding device in the method embodiment shown in the figure may include a receiving unit 701, a first decoding unit 702, an animation acquisition unit 703, a second decoding unit 704, and an image fusion unit 705. The detailed description of each unit is as follows:

[0122] The receiving unit 701 may be configured to receive a first data stream and a second data stream, wherein the first data stream is obtained by encoding the identifiers of M sign language animations, the identifiers of the M sign language animations include identifiers of sign language animations corresponding to M words, the text information of the first voice information includes a sentence to be decoded, the sentence to be decoded consists of M words, and the second data stream is obtained by encoding the first video information, and the first voice information and the first video information are acquired in the same time period;

[0123] The first decoding unit 702 may be configured to decode the identifiers of the M sign language animations to obtain the identifiers of the M sign language animations;

[0124] The animation acquisition unit 703 may be used to search for the sign language animations corresponding to the identifiers of the M sign language animations in the sign language library to obtain the M sign language animations, wherein the sign language library includes a plurality of sign language animations and the identifiers corresponding to the plurality of sign language animations;

[0125] The second decoding unit 704 may be used to decode the second data stream to obtain the first video information;

[0126] The image fusion unit 705 can be used to insert M sign language animations into the first video information to obtain the second video information. For details, see the above Figure 5 The description of S207 in the method embodiment shown is not repeated here.

[0127] In some embodiments, the first data stream further includes encoding information of the identifier of the first video information and encoding information of the indication information, and the first decoding unit 702 may also be used to decode the encoding information of the identifier of the first video information and the encoding information of the indication information to obtain the identifier and the indication information of the first video information. The identifier of the first video information may be used to indicate the video information corresponding to the identifiers of the M sign language animations, and the indication information is used to indicate the total number of image frames of the first video information.

[0128] The present application also provides a coding device, such as Figure 8 As shown, Figure 8 A structural block diagram of a coding device provided in an embodiment of the present application, the coding device 800 may be Figure 1 The encoding device in the system shown includes one or more processors 801, one or more memories 802 and a communication interface 803. The processor 801 is connected to the memory 802 and the communication interface 803 via a bus.

[0129] The communication interface 803 is used to implement information exchange between the electronic device 800 and other devices (such as decoding devices). One or more memories 802 are used to store computer instructions, and one or more processors 801 are used to execute the computer instructions stored in the one or more memories to perform the following steps:

[0130] Acquire text information of first voice information and first video information, wherein the first voice information and the first video information are acquired in the same time period, and the text information of the first voice information includes a sentence to be encoded, and the sentence to be encoded consists of M words, where M is a positive integer;

[0131] Searching for sign language animations corresponding to M words in the sign language database, and obtaining the identifiers of the M sign language animations;

[0132] Encoding the identifiers of the M sign language animations to obtain a first data stream;

[0133] Encoding the first video information to obtain a second data stream;

[0134] The first data stream and the second data stream are sent to the decoding device through the communication interface 803 .

[0135] In some embodiments, the first data stream further includes encoding information of an identifier of the first video information and encoding information of the indication information, and the processor 801 further performs encoding of the identifier of the first video information and the indication information to obtain encoding information of the identifier of the first video information and encoding information of the indication information. The identifier of the first video information can be used to indicate the video information corresponding to the identifiers of the M sign language animations, and the indication information is used to indicate the total number of image frames of the first video information.

[0136] The present application also provides a decoding device, such as Fig. 9 As shown, Fig. 9 A structural block diagram of a decoding device provided in an embodiment of the present application, the decoding device 900 may be Figure 1 The decoding device in the system shown includes one or more processors 901, one or more memories 902 and a communication interface 903. The processor 901 is connected to the memory 902 and the communication interface 903 via a bus.

[0137] The communication interface 903 is used to implement information exchange between the electronic device 900 and other devices (such as encoding devices). One or more memories 902 are used to store computer instructions, and one or more processors 901 are used to execute the computer instructions stored in the one or more memories to perform the following steps:

[0138] receiving a first data stream and a second data stream through the communication interface 903, wherein the first data stream is obtained by encoding the identifiers of M sign language animations, the identifiers of the M sign language animations include identifiers of the sign language animations corresponding to M words respectively, the text information of the first voice information includes a sentence to be decoded, the sentence to be decoded consists of M words, the second data stream is obtained by encoding the first video information, and the first voice information and the first video information are acquired in the same time period;

[0139] Decoding the first data stream to obtain identifiers of M sign language animations;

[0140] Searching for sign language animations corresponding to the identifiers of the M sign language animations in a sign language library to obtain the M sign language animations, wherein the sign language library includes a plurality of sign language animations and the identifiers corresponding to the plurality of sign language animations;

[0141] Decoding the second data stream to obtain first video information;

[0142] Insert M sign language animations into the first video information to obtain the second video information.

[0143] In some embodiments, the first data stream further includes coding information of the identifier of the first video information and coding information of the indication information, and the processor 901 further performs decoding of the coding information of the identifier of the first video information and the coding information of the indication information to obtain the identifier and the indication information of the first video information. The identifier of the first video information can be used to indicate the video information corresponding to the identifiers of the M sign language animations, and the indication information is used to indicate the total number of image frames of the first video information.

[0144] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the video encoding and decoding methods in the above embodiment and various possible implementation methods thereof are implemented.

[0145] An embodiment of the present application provides a computer program, which includes instructions. When the computer program is executed by a computer, an encoding device can execute the process executed by the encoding device in the above embodiment and its various possible implementations, or a decoding device can execute the process executed by the decoding device in the above embodiment and its various possible implementations.

[0146] An embodiment of the present application provides a chip system, which includes a processor for supporting an encoding device to implement the functions involved in the methods in the above-mentioned embodiments and their various possible modes, or a decoding device to implement the functions involved in the methods in the above-mentioned embodiments and their various possible modes.

[0147] In a possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the encoding device or the decoding device. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0148] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0149] It is to be understood that those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the various embodiments of the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0150] Those skilled in the art will appreciate that the functions described in the various illustrative logic blocks, modules, and algorithm steps disclosed in the various embodiments of the present application can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or includes any communication media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this way, computer-readable media can generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the technology described in the present application. A computer program product may include computer-readable media.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0155] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0156] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video encoding method, characterized in that: Applied to an encoding device, the method comprises: Acquire text information of first voice information and first video information, where the first voice information and the first video information are acquired in the same time period, and the text information of the first voice information includes a sentence to be encoded, where the sentence to be encoded consists of M words, where M is a positive integer; Searching for sign language animations corresponding to the M words in a sign language library to obtain identifiers of the M sign language animations, wherein the M sign language animations correspond to the M words one-to-one, and the sign language library includes a plurality of words and identifiers of the sign language animations corresponding to the plurality of words; Encoding the identifiers of the M sign language animations to obtain a first data stream; Encoding the first video information to obtain a second data stream; The first data stream and the second data stream are transmitted to a decoding device.

2. The method according to claim 1, characterized in that The identifiers of the M sign language animations are arranged in the order of the words in the sentence to be encoded.

3. The method according to claim 2, characterized in that The encoding of the identifiers of the M sign language animations to obtain a first data stream includes: For each of the M sign language animation identifiers, determine the encoding information of each identifier as the difference between each identifier and a previous identifier of each identifier; the first data stream includes the encoding information of each identifier.

4. The method according to claim 2, characterized in that The method further comprises: Obtaining an identifier of the first video information; The identifier of the first video information is encoded to obtain the encoding information of the identifier of the first video information; the first data stream also includes the encoding information of the identifier of the first video information.

5. The method according to claim 4, characterized in that The method further comprises: Acquire indication information, where the indication information is used to indicate the total number of image frames of the first video information; The indication information is encoded to obtain encoding information of the indication information; the first data stream also includes the encoding information of the indication information.

6. The method according to claim 5, characterized in that The indication information includes the duration of the first voice information and the encoding rate of the first video information, and encoding the indication information to obtain the encoding information of the indication information includes: Determine that the encoding information of the duration of the first voice information is a first difference value, wherein the first difference value is a difference between the duration of the first voice information and the duration of a previous voice information of the first voice information; the first data stream also includes the encoding information of the duration of the first voice information; The encoding information of the encoding rate of the first video information is determined as a second difference, where the second difference is the difference between the encoding rate of the first video information and the encoding rate of the previous video information of the first video information; the first data stream also includes the encoding information of the encoding rate of the first video information.

7. The method according to any one of claims 1 to 6, characterized in that: Before acquiring the text information of the first voice information and the first video information, the method further includes: receiving a second voice message and a second video message; Converting the second voice information into text to obtain text information of the second voice information; Recognize sentences in the text information of the second voice information to obtain at least one sentence; Determine the sentence to be encoded from the at least one sentence, the sentence to be encoded being one sentence; Determining that the first voice information is the voice information corresponding to the sentence to be encoded in the second voice information; Determine that the first video information is the video information corresponding to the sentence to be encoded in the second video information.

8. A video decoding method, characterized in that: Applied to a decoding device, the method comprises: receiving a first data stream and a second data stream, wherein the first data stream is obtained by encoding the identifiers of M sign language animations, the identifiers of the M sign language animations include identifiers of the sign language animations corresponding to M words respectively, the text information of the first voice information includes a sentence to be decoded, the sentence to be decoded is composed of the M words, the second data stream is obtained by encoding the first video information, and the first voice information and the first video information are acquired in the same time period; Decoding the first data stream to obtain identifiers of the M sign language animations; Searching for sign language animations corresponding to the identifiers of the M sign language animations in a sign language library to obtain M sign language animations, wherein the sign language library includes a plurality of sign language animations and the identifiers corresponding to the plurality of sign language animations; Decoding the second data stream to obtain the first video information; The M sign language animations are inserted into the first video information to obtain second video information.

9. The method according to claim 8, characterized in that The identifiers of the M sign language animations are arranged in the order of words in the sentence to be decoded, the first data stream includes encoding information of each identifier of the M sign language animations, and the encoding information of each identifier is the difference between each identifier and a previous identifier of each identifier, and the decoding of the first data stream to obtain the identifiers of the M sign language animations includes: For the encoding information of each identifier, determine the decoding information of each identifier as the sum of the difference and a previous identifier of each identifier.

10. The method according to claim 9, characterized in that The first data stream also includes coding information of an identifier of the first video information, and the decoding of the first data stream further includes: The encoded information of the identifier of the first video information is decoded to obtain the identifier of the first video information.

11. The method according to claim 10, characterized in that The first data stream further includes coding information of indication information, and the decoding of the first data stream further includes: The encoding information of the indication information is decoded to obtain the indication information, where the indication information is used to indicate the total number of image frames of the first video information.

12. The method according to claim 11, characterized in that The coding information of the indication information includes coding information of the duration of the first voice information and coding information of the coding rate of the first video information, the coding information of the duration of the first voice information is a first difference, the first difference is the difference between the duration of the first voice information and the duration of a previous voice information of the voice information, the coding information of the coding rate of the first video information is a second difference, the second difference is the difference between the coding rate of the first video information and the coding rate of a previous video information of the first video information, and the decoding of the coding information of the indication information to obtain the indication information includes: Decoding the encoding information of the duration of the first voice information to obtain the duration of the first voice information, where the duration of the first voice information is the sum of the first difference and the duration of the previous voice information; The coding information of the coding rate of the first video information is decoded to obtain the coding rate of the first video information, where the coding rate of the first video information is the sum of the coding rate of the second difference and the previous video information of the first video information.

13. The method according to any one of claims 8 to 12, characterized in that: The inserting the M sign language animations into the first video information to obtain the second video information includes: Selecting an animation image with a target number of frames from the M sign language animations to obtain a sign language animation to be inserted, wherein the target number of frames is not greater than the total number of image frames of the first video information; The i-th frame of the animation image in the M sign language animations is merged with the i-th frame of the image in the first video information to obtain the i-th frame of the image in the second video information, where i is a positive integer not greater than the total number of frames of the image of the M sign language animations.

14. The method according to claim 13, characterized in that The step of selecting an animation image with a target number of frames from the M sign language animations to obtain the sign language animation to be inserted comprises: One frame of animation image is selected from the M sign language animations every preset frame to obtain the sign language animation to be inserted, the preset frame is a positive integer not less than the quotient of the total number of image frames of the M sign language animations and the total number of image frames of the first video information, and the target number of frames is the total number of image frames of the sign language animation to be inserted.

15. A video encoding device, characterized in that: include: Memory and processor; The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to execute the method according to any one of claims 1 to 7.

16. A video decoding device, characterized in that: include: Memory and processor; The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to execute the method according to any one of claims 8 to 14.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images