A data processing method, device, computer equipment and storage medium

Abstract

The embodiment of the present application discloses a data processing method, device, computer equipment and storage medium, wherein the method includes the following steps: obtaining a target video, wherein at least one video frame among multiple video frames of the target video includes text information; determining the position information of the text information in the target video; sampling the target video using a first sampling frequency to obtain a first video frame sequence; extracting text video frame fragments from the first video frame sequence according to the position information to obtain a text video frame fragment sequence; determining the similarity between each text video frame fragment and other text video frame fragments in the text video frame fragment sequence, and obtaining a target text video frame fragment from the text video frame fragment sequence according to the similarity; obtaining the text information in the target video based on the target text video frame fragment. The present application can improve the efficiency and accuracy of lyrics detection in a video.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of mobile Internet, user-generated content (UGC) has gradually become a major component of audio and video content. On music platforms, it is necessary to identify the song IDs sung by users in UGC videos in order to better classify and recommend content. Many UGC contents incorporate user creations, and the song ID cannot be determined based solely on the UGC audio data. Therefore, the song ID is generally located by identifying the lyrics and subtitles in the video.

[0003] Most existing video subtitle localization solutions convert videos into image sequences and then detect them frame by frame for subtitle text recognition. However, due to the complexity of the video background, some image feature information in the video scene is easily misdetected as text, resulting in errors in lyrics detection. In addition, frame-by-frame detection also takes a long time to screen valid text and remove text duplication, resulting in low efficiency in lyrics detection. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, computer device, and storage medium, which can improve the efficiency and accuracy of lyrics detection in videos.

[0005] On the one hand, an embodiment of the present application provides a data processing method, which may include:

[0006] Acquire a target video, wherein at least one video frame among multiple video frames of the target video includes text information;

[0007] Determine position information of the text information in the target video, where the position information is used to indicate the same region where the at least one text information is located in each video frame;

[0008] Sampling the target video using a first sampling frequency to obtain a first video frame sequence;

[0009] Extracting text video frame segments from the first video frame sequence according to the position information to obtain a text video frame segment sequence;

[0010] Determine the similarity between each text video frame segment and other text video frame segments in the text video frame segment sequence, and obtain a target text video frame segment from the text video frame segment sequence according to the similarity;

[0011] The text information in the target video is obtained based on the target text video frame segment.

[0012] In a feasible implementation manner, determining the position information of the text information in the target video includes:

[0013] Sampling the target video using a second sampling frequency to obtain a second video frame sequence of the target video;

[0014] Acquire a binary image of the second video frame sequence based on a text detection network;

[0015] The position information of the text information in the target video is obtained according to the binary image.

[0016] In a feasible implementation manner, extracting text video frame segments from the first video frame sequence according to the position information to obtain a text video frame segment sequence includes:

[0017] Extracting the position area corresponding to the position information from each first video frame of the first video frame sequence, and generating a text video frame segment of each first video frame;

[0018] The text video frame segments of each of the first video frames are sorted into a text video frame segment sequence according to the time sequence of the text video frame segments.

[0019] In a feasible implementation manner, determining the similarity between each text video frame segment and other text video frame segments in the text video frame segment sequence includes:

[0020] The similarity between each text video frame segment except the first text video frame segment in the text video frame segment sequence and the previous text video frame segment of the text video frame segment is determined to obtain the similarity of each text video frame segment.

[0021] In a feasible implementation manner, the method further includes: obtaining a change in similarity of each of the text video frame segments based on the similarity of each of the text video frame segments;

[0022] The step of acquiring a target text video frame segment from the text video frame segment sequence according to the similarity comprises:

[0023] According to the similarity and similarity change of each of the text video frame segments, a target text video frame segment is obtained from the text video frame segment sequence.

[0024] In a feasible implementation manner, the obtaining, based on the similarity of each of the text video frame segments, a similarity change of each of the text video frame segments includes:

[0025] Generating a similarity sequence according to the similarities of each of the text video frame segments; the similarities in the similarity sequence are arranged in the time sequence of each of the text video frame segments;

[0026] A similarity change value between each similarity except the first similarity in the similarity sequence and a previous similarity of the similarity is calculated, wherein the similarity change value is used to represent a similarity change of the text video frame segment.

[0027] In a feasible implementation manner, acquiring a target text video frame segment from the text video frame segment sequence according to the similarity and similarity change of each of the text video frame segments includes:

[0028] If the similarity of the text video frame segments in the text video frame segment sequence is greater than a first threshold and the similarity change value is greater than a second threshold, the text video frame segment is determined as a target text video frame segment.

[0029] In a feasible implementation manner, the acquiring text information in the target video based on the target text video frame segment includes:

[0030] If the number of the target text video frame segments is one, extracting text information in the target text video frame segments using a text recognition method, and determining the text information in the target text video frame segments as the text information in the target video;

[0031] If the number of the target text video frame fragments is greater than one, at least two of the target text video frame fragments are spliced ​​to generate a spliced ​​image, a text recognition method is used to extract text information in the spliced ​​image, and the text information in the spliced ​​image is determined as the text information in the target video.

[0032] On the one hand, an embodiment of the present application provides a data processing device, which may include:

[0033] A video acquisition unit, used for acquiring a target video, wherein at least one video frame among multiple video frames of the target video includes text information;

[0034] A position information determining unit, used to determine the position information of the text information in the target video, wherein the position information is used to indicate the same area where the at least one text information is located in each video frame;

[0035] A first video frame acquisition unit, configured to sample the target video using a first sampling frequency to obtain a first video frame sequence;

[0036] A sequence generating unit, configured to extract text video frame segments from the first video frame sequence according to the position information to obtain a text video frame segment sequence;

[0037] a target segment acquisition unit, configured to determine the similarity between each text video frame segment and other text video frame segments in the text video frame segment sequence, and acquire a target text video frame segment from the text video frame segment sequence according to the similarity;

[0038] A text information acquisition unit is used to acquire text information in the target video based on the target text video frame segment.

[0039] In a feasible implementation manner, the location information determining unit is specifically configured to:

[0040] Sampling the target video using a second sampling frequency to obtain a second video frame sequence of the target video;

[0041] Acquire a binary image of the second video frame sequence based on a text detection network;

[0042] The position information of the text information in the target video is obtained according to the binary image.

[0043] In a feasible implementation manner, the sequence generation unit is specifically used to:

[0044] Extracting the position area corresponding to the position information from each first video frame of the first video frame sequence, and generating a text video frame segment of each first video frame;

[0045] The text video frame segments of each of the first video frames are sorted into a text video frame segment sequence according to the time sequence of the text video frame segments.

[0046] In a feasible implementation manner, the target segment acquisition unit is specifically configured to:

[0047] The similarity between each text video frame segment except the first text video frame segment in the text video frame segment sequence and the previous text video frame segment of the text video frame segment is determined to obtain the similarity of each text video frame segment.

[0048] In a feasible implementation, it also includes:

[0049] The change status acquisition unit is used to acquire the similarity change status of each of the text video frame segments based on the similarity of each of the text video frame segments.

[0050] In a feasible implementation manner, the target segment acquisition unit is specifically configured to:

[0051] According to the similarity and similarity change of each of the text video frame segments, a target text video frame segment is obtained from the text video frame segment sequence.

[0052] In a feasible implementation manner, the change situation acquisition unit is specifically used to:

[0053] Generating a similarity sequence according to the similarities of each of the text video frame segments; the similarities in the similarity sequence are arranged in the time sequence of each of the text video frame segments;

[0054] A similarity change value between each similarity except the first similarity in the similarity sequence and a previous similarity of the similarity is calculated, wherein the similarity change value is used to represent a similarity change of the text video frame segment.

[0055] In a feasible implementation manner, the target segment acquisition unit is specifically configured to:

[0056] If the similarity of the text video frame segments in the text video frame segment sequence is greater than a first threshold and the similarity change value is greater than a second threshold, the text video frame segment is determined as a target text video frame segment.

[0057] In a feasible implementation manner, the text information acquisition unit is specifically used to:

[0058] If the number of the target text video frame segments is one, extracting text information in the target text video frame segments using a text recognition method, and determining the text information in the target text video frame segments as the text information in the target video;

[0059] If the number of the target text video frame fragments is greater than one, at least two of the target text video frame fragments are spliced ​​to generate a spliced ​​image, a text recognition method is used to extract text information in the spliced ​​image, and the text information in the spliced ​​image is determined as the text information in the target video.

[0060] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the above method steps.

[0061] On the one hand, an embodiment of the present application provides a computer device, including: a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code to execute the above-mentioned method steps.

[0062] In one aspect, the present application provides a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above method steps.

[0063] In an embodiment of the present application, by acquiring a target video, at least one video frame among multiple video frames of the target video includes text information, further determining the position information of the text information in the target video, sampling the target video using a first sampling frequency to obtain a first video frame sequence, and then extracting text video frame fragments from the first video frame sequence according to the position information to obtain a text video frame fragment sequence, further determining the similarity between each text video frame fragment and other text video frame fragments in the text video frame fragment sequence, obtaining a target text video frame fragment from the text video frame fragment sequence according to the similarity, and finally obtaining the text information in the target video based on the target text video frame fragment. With the above method, only the features of the area corresponding to the position information need to be detected, and there is no need to detect the target video frame by frame, thereby avoiding the problem of misdetecting the features in the video background as text, and the serious time-consuming problem of text screening and text deduplication, and improving the efficiency and accuracy of lyrics detection in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 This is a network architecture diagram of data processing provided by an embodiment of the present application;

[0066] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0067] Figure 3 This is an example schematic diagram of determining location information provided by an embodiment of the present application;

[0068] Figure 4 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0069] Figure 5a is an example schematic diagram of a text video frame segment sequence provided in an embodiment of the present application;

[0070] Figure 5b is an example schematic diagram of similarity provided in an embodiment of the present application;

[0071] Figure 5c is an example schematic diagram of a spliced ​​picture provided in an embodiment of the present application;

[0072] Figure 6 This is an example schematic diagram of extracting lyrics and subtitles provided in an embodiment of the present application;

[0073] Figure 7 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0074] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0076] See also Figure 1 , Figure 1 It is a network architecture diagram of data processing provided by an embodiment of the present application. The network architecture diagram may include a service server 100 and a user terminal cluster, and the user terminal cluster may include a user terminal 10a, a user terminal 10b, ..., a user terminal 10c, wherein a communication connection may exist between the user terminal clusters, for example, a communication connection exists between the user terminal 10a and the user terminal 10b, and a communication connection exists between the user terminal 10b and the user terminal 10c, and any user terminal in the user terminal cluster may have a communication connection with the service server 100, for example, a communication connection exists between the user terminal 10a and the service server 100, and a communication connection exists between the user terminal 10b and the service server 100.

[0077] Among them, the above-mentioned user terminal cluster (also including the above-mentioned user terminal 10a, user terminal 10b and user terminal 10c) can all be integrated with the target application. Optionally, the target application may include an application with the function of displaying data information such as text, images and videos. In a feasible implementation, the user terminal obtains the target video, and at least one video frame among the multiple video frames of the target video includes text information. The user terminal further determines the position information of the text information in the target video, and samples the target video using a first sampling frequency to obtain a first video frame sequence. Then, the user terminal extracts text video frame fragments from the first video frame sequence according to the position information to obtain a text video frame fragment sequence. Further, the user terminal determines the similarity between each text video frame fragment and other text video frame fragments in the text video frame fragment sequence, and obtains a target text video frame fragment from the text video frame fragment sequence according to the similarity. Finally, the user terminal obtains the text information in the target video based on the target text video frame fragment. Optionally, the above-mentioned user terminal can be in the above-mentioned Figure 1 Any user terminal selected from the user terminal cluster of the corresponding embodiment, for example, the user terminal may be the above-mentioned user terminal 10b.

[0078] It is understandable that the method provided in the embodiment of the present application can be executed by a computer device, and the computer device includes but is not limited to a terminal or a server. The business server 100 in the embodiment of the present application can be a computer device, and the user terminal in the user terminal cluster can also be a computer device, which is not limited here. The above-mentioned business server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The above-mentioned terminals may include: smart phones, tablet computers, laptops, desktop computers, smart TVs, smart speakers, desktop computers, smart watches and other smart terminals with image recognition functions, but are not limited to this. Among them, the user terminal and the business server can be directly or indirectly connected by wired or wireless communication, and this application is not limited here.

[0079] For further understanding, please see Figure 2 , Figure 2 is a flow chart of a data processing method provided by an embodiment of the present application. The method may be performed by a user terminal (for example, the above Figure 1 The user terminal shown in the figure) can also be executed by the user terminal and the service server (as shown in the figure above) Figure 1The data processing method is executed by the service server 100 in the corresponding embodiment. For ease of understanding, this embodiment takes the method executed by the above-mentioned user terminal as an example. Among them, the data processing method may at least include the following steps S101-S106:

[0080] S101, obtaining a target video.

[0081] Specifically, the user terminal obtains the target video. It can be understood that at least one video frame among the multiple video frames of the target video includes text information. The target video can be a video stored locally in the user terminal, and the user terminal can directly obtain it locally. The target video can also be a video stored in the server of each website platform on the Internet, and the user terminal can download it through the network. For example, the target video can be a music video (Music Video, MV) of a song on a music platform, and the text information is the lyrics and subtitles of the song in the MV.

[0082] S102: Determine the position information of the text information in the target video.

[0083] Specifically, the position information is the area where the text information appears in the target video, and the position coordinates of the above area are the position information of the text information. It should be noted that the text information contained in the target video of the embodiment of the present application has certain characteristics, that is, the position of the text information in each video frame of the target video is roughly the same, for example, the lyrics and subtitles in the music video are generally at the bottom of each video frame. Therefore, the position information of the text information determined in this step is not the specific position of each text information in each video frame, but represents the approximate common position of the text information contained in each video frame of the target video.

[0084] The following is an explanation using a song MV. The text information is the lyrics and subtitles of the song. In the MV, the lyrics and subtitles usually appear at a fixed position below the video. Therefore, a small number of video frames can be extracted from the MV, and the location information of the text information can be determined based on the area of ​​the lyrics and subtitles in the extracted video frames. Specifically, the specific steps for determining the location information are as follows:

[0085] The target video is sampled with a second sampling frequency to obtain a second video frame of the target video. The second sampling frequency is a pre-set sampling frequency. Since the number of second video frames required is small, the second sampling frequency can be set to a larger value. In a specific embodiment, in order to improve the positioning accuracy of the position information, the sampled second video frame can be a key image frame in the target video. Furthermore, based on a text detection network, a binary image of the second video frame is obtained, and the position information of the text information in the target video is obtained according to the binary image. The binary image is used to identify the text information in the first video frame. The text detection network can use a neural network, specifically a VGG16 network architecture. The acquisition of the binary image is described below using the VGG16 network architecture.

[0086] See also Figure 3 , Figure 3 It is an example schematic diagram of determining position information provided by an embodiment of the present application. Specifically, the second video frame is input into the VGG16 network, and the network outputs two probability maps of the same size as the second video frame, one of which represents the probability that each pixel belongs to a text, and the other represents the probability that each pixel belongs to between two texts. Combined with the two probability maps, a binary map of the same size as the second video frame is generated, wherein the white area in the binary map represents that the area is identified with text. After obtaining the binary maps of all second video frames according to the network, the number of times each pixel is judged as text information is counted between frames. In order to reduce the misjudgment of text information, a number threshold is usually set to determine the text information. For example, the number threshold can be set to 2. If the number of times a pixel is judged as text information is greater than the number threshold, the pixel is determined as text information. Since there may be interference information in the background of the target video, the above-mentioned text information can have multiple areas, and the above-mentioned text information can be further screened to obtain the text information corresponding to the lyrics and subtitles. For example, the text area is screened by the following rules: 1) The area is greater than 150 pixels, 2) The text information is at the bottom of the video frame. Finally, the position information of the filtered text information is obtained, that is, the positioning of the text information is completed. Generally, the text information corresponds to a rectangular area, and the position information can be expressed by the coordinates of the upper left corner of the rectangular area corresponding to the text information and the width and height of the rectangular area. The horizontal and vertical coordinates of the upper left corner coordinates and the width and height of the rectangular area are represented by letters x, y, w, and h, respectively, that is, the rectangular area is (x, y, w, h).

[0087] S103: Sample the target video using a first sampling frequency to obtain a first video frame sequence.

[0088] Specifically, the user terminal samples the target video using a first sampling frequency to obtain a first video frame sequence. It can be understood that the first sampling frequency is a preset sampling frequency. For example, the first sampling frequency can be set to 1 Hz, that is, one frame I is sampled per second for the target video. n , where the subscript n represents the sampling frame of the nth second, I n It should be noted that a large number of first video frames are obtained by using a first sampling frequency, and the first sampling frequency is greater than the second sampling frequency.

[0089] S104, extracting text video frame segments from the first video frame sequence according to the position information to obtain a text video frame segment sequence;

[0090] Specifically, the user terminal extracts text video frame segments from the first video frame sequence according to the position information to obtain a text video frame segment sequence, wherein the position area is the area where the text information is located, for example, if the text information is lyrics and subtitles, the position area is a rectangular area corresponding to the lyrics and subtitles. Further, the text video frame segments of each first video frame corresponding to the target video are determined as a text video frame segment sequence, and the text video frame segments in the text video frame segment sequence are arranged in the time order of the text video frame segments. For example, if a target video with a length of 1 minute is sampled at one frame per second, then sampling is performed at the time of t=1s, 2s, 3s...60s to generate 60 first video frames, and further 60 text video frame segments are generated. Then, the first text video frame segment in the text video frame segment sequence is the text video frame segment corresponding to the time of t=1s in the target video, the second text video frame segment is the text video frame segment corresponding to the time of t=2s in the target video, and the last text video frame segment is the text video frame segment corresponding to the time of t=60s in the target video.

[0091] It should be noted that in order to improve robustness and avoid truncation of text in the edge area of ​​the text information, the edges of the upper, lower, left and right sides of the rectangular area corresponding to the text information can be increased by 10% in length respectively. For example, if the rectangular area is (x, y, w, h), after adjustment, the coordinates of the upper left vertex of the rectangular area are (x-0.1*w, y-0.1*h), the coordinates of the lower right vertex are (x+1.1*w, y+1.1*h), the width of the rectangular area is 1.2*w, and the length is 1.2*h.

[0092] S105, determining the similarity between each text video frame segment and other text video frame segments in the text video frame segment sequence, and acquiring a target text video frame segment from the text video frame segment sequence according to the similarity.

[0093] Specifically, the user terminal obtains the similarity corresponding to the text video frame fragment based on the text video frame fragment sequence, and the similarity is used to measure the similarity between the text information in the two text video frame fragments. The similarity is the similarity between the text video frame fragment in the text video frame fragment sequence and its previous text video frame fragment. Specifically, the Dice coefficient can be used as the similarity. When the text information in the two text video frame fragments is exactly the same, the Dice coefficient is 1, and when the text information in the two text video frame fragments is completely different, the Dice coefficient is 0. Furthermore, according to the similarity, the repeated text video frame fragments are deleted from the text video frame fragment sequence to obtain the target text video frame fragment.

[0094] S106: Acquire text information in the target video based on the target text video frame segment.

[0095] Specifically, the user terminal obtains the text information in the target video based on the target text video frame fragment. It can be understood that the target text video frame fragment obtained from the text video frame fragment sequence can be one or more. If the number of the target text video frame fragments is one, a text recognition method is used to extract the text information in the target text video frame fragment, and the text information in the target text video frame fragment is determined as the text information in the target video. If the number of the target text video frame fragments is greater than one, the at least two target text video frame fragments are spliced ​​to generate a spliced ​​picture, and a text recognition method is used to extract the text information in the spliced ​​picture, and the text information in the spliced ​​picture is determined as the text information in the target video. The text recognition method can specifically be OCR.

[0096] In an embodiment of the present application, by acquiring a target video, at least one video frame among multiple video frames of the target video includes text information, further determining the position information of the text information in the target video, sampling the target video using a first sampling frequency to obtain a first video frame sequence, and then extracting text video frame fragments from the first video frame sequence according to the position information to obtain a text video frame fragment sequence, further determining the similarity between each text video frame fragment and other text video frame fragments in the text video frame fragment sequence, obtaining a target text video frame fragment from the text video frame fragment sequence according to the similarity, and finally obtaining the text information in the target video based on the target text video frame fragment. With the above method, only the features of the area corresponding to the position information need to be detected, and there is no need to detect the target video frame by frame, thereby avoiding the problem of misdetecting the features in the video background as text, and the serious time-consuming problem of text screening and text deduplication, and improving the efficiency and accuracy of lyrics detection in the video.

[0097] See also Figure 4 , Figure 4 is a flow chart of a data processing method provided by an embodiment of the present application. The method may be performed by a user terminal (for example, the above Figure 1 The user terminal shown in the figure) can also be executed by the user terminal and the service server (as shown in the figure above) Figure 1 The data processing method is executed by the service server 100 in the corresponding embodiment. For ease of understanding, this embodiment takes the method executed by the above-mentioned user terminal as an example. Among them, the data processing method may at least include the following steps S201-S208:

[0098] S201, obtaining a target video.

[0099] Wherein, step S201 of the embodiment of the present invention refers to Figure 2 The specific description of step S101 of the illustrated embodiment is not repeated here.

[0100] S202: Determine the position information of the text information in the target video.

[0101] Wherein, step S202 of the embodiment of the present invention refers to Figure 2 The specific description of step S102 of the illustrated embodiment is not repeated here.

[0102] S203: Sample the target video using a first sampling frequency to obtain a first video frame sequence.

[0103] Wherein, step S203 of the embodiment of the present invention refers to Figure 2 The specific description of step S103 of the illustrated embodiment is not repeated here.

[0104] S204: extract text video frame segments from the first video frame sequence according to the position information to obtain a text video frame segment sequence.

[0105] Wherein, step S204 of the embodiment of the present invention refers to Figure 2 The specific description of step S104 of the illustrated embodiment is not repeated here.

[0106] S205, determining the similarity between each text video frame segment except the first text video frame segment in the text video frame segment sequence and the previous text video frame segment of the text video frame segment, and obtaining the similarity of each text video frame segment;

[0107] Specifically, the user terminal obtains a text video frame segment in a text video frame segment sequence, where the text video frame segment is any text video frame segment except the first text video frame segment in the text video frame segment sequence, calculates the similarity between a previous text video frame segment of the text video frame segment and the text video frame segment, and determines the similarity as the similarity corresponding to the text video frame segment.

[0108] Combine the following Figure 5a-Figure 5b The acquisition of similarity is described below, wherein the similarity adopts the Dice coefficient. Figure 5a is an example schematic diagram of a text video frame segment sequence provided in an embodiment of the present application, Figure 5b is an example diagram of similarity provided in the embodiment of the present application. Figure 5a The figure includes 15 text video frame segments arranged in chronological order, numbered 1-15, each of which includes text information, which is the lyrics and subtitles in the MV. In order to improve the accuracy of similarity, edge detection algorithms are generally used to obtain edge maps of text video frame segments, and morphological dilation operations are further used to thicken the single-pixel edges of the edge maps. See Figure 5b , the figure shows a text video frame segment after edge detection and morphological dilation operations. When calculating the Dice coefficient of the text video frame segment, only the white subtitle area in the text video frame segment needs to be paid attention to, and the black background area does not need to be paid attention to. Specifically, each text video frame segment is selected from the text video frame segments numbered 2-15, and the Dice coefficient between the previous text video frame segment of the text video frame segment and the text video frame segment is calculated, and the Dice coefficient is determined as the Dice coefficient corresponding to the text video frame segment. As shown in the figure, the Dice coefficients corresponding to the text video frame segments numbered 2-15 are "0.49, 0.61, ... 1.00".

[0109] S206: Based on the similarities of the text video frame segments, obtain similarity changes of the text video frame segments.

[0110] Specifically, the user terminal generates a similarity sequence according to the similarities corresponding to the text video frame segments, and the similarities in the similarity sequence are arranged in the time sequence of the text video frame segments, see Figure 5b, the Dice coefficient is used to illustrate the similarity sequence. The Dice coefficients "0.49, 0.61, ... 1.00" corresponding to the text video frame segments numbered 2-15 are the similarity sequences corresponding to the text video frame segments. Further, the similarities in the similarity sequence are obtained, and the similarity change between the previous similarity of the similarity and the similarity is calculated, and the similarity change is determined as the similarity change corresponding to the text video frame segment. The similarity change can specifically be the difference between the similarity and the previous similarity of the similarity. The similarity is any similarity in the similarity sequence except the first similarity. Figure 5b As shown, each similarity is selected from the similarities corresponding to the text video frame segments numbered 3-15, and the difference between the similarity and the previous similarity of the similarity is calculated, and the difference is determined as the similarity change value corresponding to the text video frame segment. As shown in the figure, the similarity change value corresponding to the text video frame segment numbered 3-15 is "0.11, 0.11, ... 0.84".

[0111] S207: acquiring a target text video frame segment from the text video frame segment sequence according to the similarity and similarity change of each of the text video frame segments.

[0112] Specifically, the user terminal obtains a text video frame segment from the text video frame segment sequence, and the text video frame segment is any text video frame segment in the text video frame segment sequence except the first text video frame segment and the second text video frame segment; if the similarity of the text video frame segment is greater than a first threshold, and the similarity change value of the text video frame segment is greater than a second threshold, the text video frame segment is determined as a target text video frame segment, and the first threshold and the second threshold are preset. Specifically, the first threshold can be set to 0.8, and the second threshold can be set to 0.1, that is, the text video frame segment with a similarity greater than 0.8 and a similarity change value greater than 0.1 is the target text video frame segment. Please refer to Figure 5b , the similarity corresponding to the text video frame segment sequence is "0.49, 0.61, ... 1.00", and the similarity change value is "0.11, 0.11, ... 0.84". The text video frame segments that meet the first threshold and the second threshold conditions are numbered 5, 7, 13 and 15, that is, the text video frame segments numbered 5, 7, 13 and 15 are the target text video frame segments.

[0113] S208: Acquire text information in the target video based on the target text video frame segment.

[0114] Specifically, the user terminal obtains the text information in the target video based on the target text video frame segment. It can be understood that the target text video frame segment obtained from the text video frame segment sequence can be one or more. If the number of the target text video frame segment is one, a text recognition method will be used to extract the text information in the target text video frame segment, and the text information in the target text video frame segment will be determined as the text information in the target video. If the number of the target text video frame segment is greater than one, the at least two target text video frame segments will be spliced ​​to generate a spliced ​​image, and a text recognition method will be used to extract the text information in the spliced ​​image, and the text information in the spliced ​​image will be determined as the text information in the target video. The text recognition method can specifically be OCR. The following is an explanation of the case of multiple target text video frame segments. Please refer to Figure 5b and 5c , Figure 5c is an example schematic diagram of a spliced ​​picture provided in an embodiment of the present application. Figure 5b It can be seen that the target text video frame segments include 4, so the above 4 target text video frame segments are spliced ​​to generate a spliced ​​image, such as Figure 5c As shown, a text recognition method is further used to extract text information in the spliced ​​image, and the above text information is the text information in the target video.

[0115] Combine the following Figure 6 The extraction of lyrics subtitles from the target video is described. Figure 6 1 is an example schematic diagram of extracting lyrics and subtitles provided in an embodiment of the present application. Figure 6 As shown, the target video is the MV of a song, and the target video includes lyrics and subtitles. First, a small number of video frames are sampled and extracted from the target video, and the position information of the lyrics and subtitles in the target video is located according to the above video frames; the target video is further sampled to obtain the first video frame in the target video, and the lyrics and subtitles area in the first video frame is extracted according to the position information to generate a text video frame segment of the first video frame; based on all the text video frame segments of the first video frame, a text video frame segment sequence corresponding to the target video is generated, and further, the similarity between the text video frame segments is calculated according to the Dice coefficient, and the text video frame segment sequence is deduplicated according to the Dice coefficient to obtain the target text video frame segment in the text video frame segment sequence; finally, the text information in the target text video frame segment is obtained through a text recognition method.

[0116] In an embodiment of the present application, by acquiring a target video, at least one video frame among multiple video frames of the target video includes text information, further determining the position information of the text information in the target video, sampling the target video using a first sampling frequency to obtain a first video frame sequence, and then extracting text video frame fragments from the first video frame sequence according to the position information to obtain a text video frame fragment sequence, further determining the similarity between each text video frame fragment and other text video frame fragments in the text video frame fragment sequence, obtaining a target text video frame fragment from the text video frame fragment sequence according to the similarity, and finally obtaining the text information in the target video based on the target text video frame fragment. With the above method, only the features of the area corresponding to the position information need to be detected, and there is no need to detect the target video frame by frame, thereby avoiding the problem of misdetecting the features in the video background as text, and the serious time-consuming problem of text screening and text deduplication, and improving the efficiency and accuracy of lyrics detection in the video.

[0117] See also Figure 7 , Figure 7 : is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device may be a computer program (including program code) running in a computer device, for example, the data processing device is an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Figure 7 As shown, the data processing device 1 of the embodiment of the present application may include: a video acquisition unit 11, a position information determination unit 12, a first video frame acquisition unit 13, a sequence generation unit 14, a target segment acquisition unit 15, and a text information acquisition unit 16.

[0118] A video acquisition unit 11 is used to acquire a target video, wherein at least one video frame among multiple video frames of the target video includes text information;

[0119] A position information determining unit 12, used to determine the position information of the text information in the target video, wherein the position information is used to indicate the same area where the at least one text information is located in each video frame;

[0120] A first video frame acquisition unit 13, configured to sample the target video using a first sampling frequency to obtain a first video frame sequence;

[0121] A sequence generating unit 14 is used to extract text video frame segments from the first video frame sequence according to the position information to obtain a text video frame segment sequence;

[0122] A target segment acquisition unit 15 is used to determine the similarity between each text video frame segment and other text video frame segments in the text video frame segment sequence, and acquire a target text video frame segment from the text video frame segment sequence according to the similarity;

[0123] The text information acquisition unit 16 is configured to acquire text information in the target video based on the target text video frame segment.

[0124] In a feasible implementation manner, the location information determining unit 12 is specifically configured to:

[0125] Sampling the target video using a second sampling frequency to obtain a second video frame sequence of the target video;

[0126] Acquire a binary image of the second video frame sequence based on a text detection network;

[0127] The position information of the text information in the target video is obtained according to the binary image.

[0128] In a feasible implementation manner, the sequence generation unit 14 is specifically used for:

[0129] Extracting the position area corresponding to the position information from each first video frame of the first video frame sequence, and generating a text video frame segment of each first video frame;

[0130] The text video frame segments of each of the first video frames are sorted into a text video frame segment sequence according to the time sequence of the text video frame segments.

[0131] In a feasible implementation manner, the target segment acquisition unit 15 is specifically configured to:

[0132] The similarity between each text video frame segment except the first text video frame segment in the text video frame segment sequence and the previous text video frame segment of the text video frame segment is determined to obtain the similarity of each text video frame segment.

[0133] See also Figure 7 , the data processing device 1 of the embodiment of the present application may also include: a change situation acquisition unit 17.

[0134] The change status acquisition unit 17 is used to acquire the similarity change status of each of the text video frame segments based on the similarity of each of the text video frame segments.

[0135] In a feasible implementation manner, the target segment acquisition unit 15 is specifically configured to:

[0136] According to the similarity and similarity change of each of the text video frame segments, a target text video frame segment is obtained from the text video frame segment sequence.

[0137] In a feasible implementation manner, the change situation obtaining unit 17 is specifically used to:

[0138] Generating a similarity sequence according to the similarities of each of the text video frame segments; the similarities in the similarity sequence are arranged in the time sequence of each of the text video frame segments;

[0139] A similarity change value between each similarity except the first similarity in the similarity sequence and a previous similarity of the similarity is calculated, wherein the similarity change value is used to represent a similarity change of the text video frame segment.

[0140] In a feasible implementation manner, the target segment acquisition unit 15 is specifically configured to:

[0141] If the similarity of the text video frame segments in the text video frame segment sequence is greater than a first threshold and the similarity change value is greater than a second threshold, the text video frame segment is determined as a target text video frame segment.

[0142] In a feasible implementation manner, the text information acquisition unit 16 is specifically used for:

[0143] If the number of the target text video frame segments is one, extracting text information in the target text video frame segments using a text recognition method, and determining the text information in the target text video frame segments as the text information in the target video;

[0144] If the number of the target text video frame fragments is greater than one, at least two of the target text video frame fragments are spliced ​​to generate a spliced ​​image, a text recognition method is used to extract text information in the spliced ​​image, and the text information in the spliced ​​image is determined as the text information in the target video.

[0145] In an embodiment of the present application, by acquiring a target video, at least one video frame among multiple video frames of the target video includes text information, further determining the position information of the text information in the target video, sampling the target video using a first sampling frequency to obtain a first video frame sequence, and then extracting text video frame fragments from the first video frame sequence according to the position information to obtain a text video frame fragment sequence, further determining the similarity between each text video frame fragment and other text video frame fragments in the text video frame fragment sequence, obtaining a target text video frame fragment from the text video frame fragment sequence according to the similarity, and finally obtaining the text information in the target video based on the target text video frame fragment. With the above method, only the features of the area corresponding to the position information need to be detected, and there is no need to detect the target video frame by frame, thereby avoiding the problem of misdetecting the features in the video background as text, and the serious time-consuming problem of text screening and text deduplication, and improving the efficiency and accuracy of lyrics detection in the video.

[0146] See also Figure 8 , Figure 8 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 8 As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a random access memory (Random Access Memory, RAM) or a non-volatile memory (non-volatile memory, NVM), such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Figure 8 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data processing application program.

[0147] exist Figure 8 In the computer device 1000 shown in the figure, the network interface 1004 can provide a network communication function, the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the data processing application stored in the memory 1005 to implement the above Figure 2-Figure 6The description of the data processing method in any corresponding embodiment will not be repeated here.

[0148] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 2-Figure 6 The description of the data processing method in any corresponding embodiment can also be performed as described above. Figure 7 The description of the data processing device in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0149] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the data processing device mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 2-Figure 6 The description of the data processing method in any corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, program instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network. Multiple computing devices distributed at multiple locations and interconnected by a communication network can constitute a blockchain system.

[0150] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned computer-readable storage medium can be a data processing device provided by any of the above-mentioned embodiments or an internal storage unit of the above-mentioned device, such as a hard disk or memory of an electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the electronic device. The above-mentioned computer-readable storage medium can also include a disk, an optical disk, a read-only memory (ROM) or a random access memory, etc. Further, the computer-readable storage medium can also include both the internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and quantities required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0151] The terms "first", "second", etc. in the claims, specification and drawings of the present invention are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Displaying the phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and / or" used in the specification and the appended claims of the present invention refers to any combination of one or more of the items listed in association and all possible combinations, and includes these combinations.

[0152] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0153] Each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Acquire a target video, wherein at least one video frame among multiple video frames of the target video includes text information; Determine position information of the text information in the target video, the position information is used to indicate the same region where at least one piece of text information is located in each video frame; the position information is determined based on a second video frame sequence in the target video; Sampling the target video using a first sampling frequency to obtain a first video frame sequence; Extracting text video frame segments from the first video frame sequence according to the position information to obtain a text video frame segment sequence; the text video segment is a segment containing text information in a video frame; thickening single-pixel edges in edge maps of text video segments in the sequence of text video frame segments; Determine the similarity between each text video frame segment and other text video frame segments in the text video frame segment sequence according to each text video segment after the single pixel edge is thickened, and obtain the similarity change of each text video frame segment based on the similarity of each text video frame segment; If the similarity of the text video frame segments in the text video frame segment sequence is greater than a first threshold and the similarity change value is greater than a second threshold, determining the text video frame segment as a target text video frame segment; The text information in the target video is obtained based on the target text video frame segment.

2. The method according to claim 1, characterized in that The determining the position information of the text information in the target video includes: Sampling the target video using a second sampling frequency to obtain a second video frame sequence of the target video; Acquire a binary image of the second video frame sequence based on a text detection network; The position information of the text information in the target video is obtained according to the binary image.

3. The method according to claim 1, characterized in that The step of extracting text video frame segments from the first video frame sequence according to the position information to obtain a text video frame segment sequence includes: Extracting the position area corresponding to the position information from each first video frame of the first video frame sequence, and generating a text video frame segment of each first video frame; The text video frame segments of each of the first video frames are sorted into a text video frame segment sequence according to the time sequence of the text video frame segments.

4. The method according to claim 1, characterized in that: The step of determining the similarity between each text video frame segment and other text video frame segments in the text video frame segment sequence according to each text video segment after the single pixel edge is thickened includes: Determine the similarity between each text video frame segment after the single pixel edge is thickened except the first text video frame segment in the text video frame segment sequence and the previous text video frame segment after the single pixel edge corresponding to the text video frame segment is thickened, and obtain the similarity of each text video frame segment.

5. The method according to claim 1, characterized in that The obtaining, based on the similarity of each of the text video frame segments, a change in the similarity of each of the text video frame segments comprises: Generating a similarity sequence according to the similarities of each of the text video frame segments; the similarities in the similarity sequence are arranged in the time sequence of each of the text video frame segments; A similarity change value between each similarity except the first similarity in the similarity sequence and a previous similarity of the similarity is calculated, wherein the similarity change value is used to represent a similarity change of the text video frame segment.

6. The method according to claim 1, characterized in that The acquiring text information in the target video based on the target text video frame segment includes: If the number of the target text video frame segments is one, extracting text information in the target text video frame segments using a text recognition method, and determining the text information in the target text video frame segments as the text information in the target video; If the number of the target text video frame fragments is greater than one, at least two of the target text video frame fragments are spliced ​​to generate a spliced ​​image, a text recognition method is used to extract text information in the spliced ​​image, and the text information in the spliced ​​image is determined as the text information in the target video.

7. A data processing device, characterized in that: include: A video acquisition unit, used for acquiring a target video, wherein at least one video frame among multiple video frames of the target video includes text information; A position information determination unit, used to determine the position information of the text information in the target video, wherein the position information is used to indicate the same area where at least one piece of text information is located in each video frame; the position information is determined based on a second video frame sequence in the target video; A first video frame acquisition unit is used to sample the target video using a first sampling frequency to obtain a first video frame sequence; the text video segment is a segment containing text information in the video frame; A sequence generating unit, configured to extract text video frame segments from the first video frame sequence according to the position information to obtain a text video frame segment sequence; a target segment acquisition unit, configured to thicken the single-pixel edge in the edge map of the text video segment in the text video frame segment sequence; determine the similarity between each text video frame segment in the text video frame segment sequence and other text video frame segments according to each text video segment after the single-pixel edge is thickened, and obtain the similarity change of each text video frame segment based on the similarity of each text video frame segment; if the similarity of the text video frame segments in the text video frame segment sequence is greater than a first threshold value and the similarity change value is greater than a second threshold value, determine the text video frame segment as a target text video frame segment; A text information acquisition unit is used to acquire text information in the target video based on the target text video frame segment.

8. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program codes, and the processor is used to call the program codes to execute the method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 6.

Images