Video line generation method and device, electronic equipment and storage medium

Abstract

The invention provides a video line generation method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining an offset type of a first line of an original video and an offset type of a second video relative to the original video and a second line of the second video; the first video is obtained after the original video is reconverted, and the second video is generated based on a video clip, corresponding to the target time point, of the first video; based on the target time point location and the offset type, intercepting a reference line from the first line; and filtering the second lines according to the reference lines and the offset type to obtain target lines of the second video. Therefore, the reference line is intercepted from the first line generated based on the original video, then the second line is corrected based on the reference line, and the reference line of the original video guides the second video to generate the target line, so that the offset between the second video and the original video is eliminated, the recognition accuracy of the target line is improved, and the recognition efficiency of the target line is improved. And subsequent query and use are facilitated.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for generating video dialogues. Background Art

[0002] On the video platform, long videos can be promoted and disseminated by generating short videos by extracting part of the content from the long videos. In the process of generating short videos, it is necessary to rely on the stored lines of the long videos to guide the production of the lines of the short videos, so as to record the lines of the short videos for subsequent query and use.

[0003] Specifically, we can first count the screen area where the lines are located in the long video, that is, the long video line area. Then, based on the correspondence between the time points of the short video and the long video, and taking the long video line area as a reference, we can determine the screen area where the lines are located in the short video as the short video line area. Then, by performing text detection, tracking and filtering on the short video line area, we can obtain the short video lines.

[0004] However, in some cases, long videos will be re-transmitted, such as Figure 1 As shown, after re-converting, its length and size may change to a certain extent, resulting in a time or position offset between the re-converted long video and the stored long video lines. If a short video is captured based on the re-converted long video at this time, the lines in the short video will also be offset from the stored long video lines. Therefore, guiding the production of short video lines based on the stored long video lines will result in a low accuracy rate in short video line recognition. Summary of the invention

[0005] In order to solve the above technical problems, the present application shows a method, device, electronic device and storage medium for generating video lines, so as to at least solve the problem in the related art that the short video captured based on the re-converted long video has a time offset with the stored long video lines, so that the short video lines are guided by the stored long video lines, which will lead to a low accuracy rate of short video lines recognition. The technical solutions disclosed in the present application are as follows:

[0006] In a first aspect, the present application provides a method for generating video lines, the method comprising:

[0007] Obtaining a first line of an original video, an offset type of a second video compared to the original video, and a second line of the second video; the first video is obtained after re-transmitting the original video, and the second video is generated based on a video segment of the first video corresponding to a target time point;

[0008] Based on the target time point and the offset type, extracting a reference line from the first line;

[0009] The second lines are filtered according to the reference lines and the offset type to obtain target lines of the second video.

[0010] In a second aspect, an embodiment of the present invention provides a video dialogue generation device, comprising:

[0011] an acquisition module, configured to acquire a first line of an original video, an offset type of a second video compared to the original video, and a second line of the second video; the first video is obtained after re-transmitting the original video, and the second video is generated based on a video segment of the first video corresponding to a target time point;

[0012] An interception module, configured to intercept reference lines from the first lines based on the target time point and the offset type;

[0013] A filtering module is used to filter the second lines according to the reference lines and the offset type to obtain target lines of the second video.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0015] processor;

[0016] a memory for storing instructions executable by the processor;

[0017] The processor is configured to execute the instructions to implement any of the above-mentioned methods for generating video lines.

[0018] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of a video dialogue generating electronic device, the video dialogue generating electronic device can execute any of the above-mentioned video dialogue generating methods.

[0019] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements any of the above-mentioned methods for generating video lines when executed by a processor.

[0020] Compared with the prior art, this application has the following advantages:

[0021] The original lines of the first video, the offset type of the second video compared to the original video, and the second lines of the second video are obtained; the first video is obtained after re-transmitting the original video, and the second video is generated based on the video clip of the first video corresponding to the target time point; based on the target time point and the offset type, the reference lines are intercepted from the first lines; according to the reference lines and the offset type, the second lines are filtered to obtain the target lines of the second video.

[0022] In this way, when the first video after the original video is re-converted changes compared to the original video, resulting in the second video generated based on the first video being offset relative to the original video, when producing the target lines of the second video, first, according to the target time point corresponding to the second video and the offset type of the second video compared to the original video, the reference lines are intercepted from the first lines produced based on the original video, and then, the second lines are corrected based on the reference lines and the offset type, and the production of the target lines for the second video is guided by the reference lines of the original video, thereby eliminating the offset between the second video and the original video, and finally obtaining a more accurate target line that matches the second video, thereby improving the recognition accuracy of the target lines and facilitating subsequent query and use. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of a video re-conversion and short video generation of the present application;

[0024] Figure 2 It is a flowchart of the steps of a video dialogue generation method of the present application;

[0025] Figure 3 is a flowchart of another method for generating video lines in the present application;

[0026] Figure 4 It is a structural block diagram of a video dialogue generation device of the present application;

[0027] Figure 5 is a schematic diagram of an electronic device of the present application;

[0028] Figure 6 It is a block diagram of a device for generating video lines in the present application. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0031] On the video platform, a second video can be generated by intercepting the content of the target time point in the original video with a longer duration. Since the second video is shorter, has a smaller amount of data, and highlights the key points, it is relatively simple and easy to spread the second video. Furthermore, the spread of the second video also has a certain promotional effect on the original video.

[0032] In the process of generating the second video, it is usually necessary to rely on the first lines of the stored original video. Based on the target time points of the second video relative to the original video, the first lines corresponding to these target time points are determined, and then the line production of the second video is guided to finally obtain the target lines of the second video.

[0033] However, in some cases, the original video will be re-rotated, and after the re-rotation, the original video will be replaced by the first video. The length of the first video relative to the original video may change to a certain extent. In this case, if the second video is captured at this time, the second video is generated based on the target time point of the first video. If there is a time offset between the first video and the original video, there will also be a time offset between the second video and the original video. Therefore, the target lines determined with reference to the first lines at the target time point are inaccurate.

[0034] Based on this, the present application provides a method for generating video lines, which can solve the above-mentioned problems, improve the accuracy of target line recognition in the second video, and facilitate subsequent query and use.

[0035] Specifically, the present application can be applied to a server or a terminal, and the server can be a local server, other servers in the network, etc. The above-mentioned terminal can be a terminal device in various forms such as a mobile phone, a tablet computer, a desktop computer, a portable notebook computer, etc., and the embodiments of the present application are not limited to this.

[0036] The above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0037] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0038] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system. Blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0039] The following is an explanation of the relevant terms mentioned in this application:

[0040] Video re-rendering: refers to the process in which a video is played, with the addition or reduction of advertisements, or the change of some content, which will cause a small change in the length or size of the video, but the video screen content will not change. For example, the original video has black borders at the top and bottom, which are removed during the re-rendering process, etc. The video re-rendering process will cause the time point of the lines to change, or the position of the lines will change relative to the entire video as the video size is re-rendered. It should be noted that for any re-rendering, the text content of the lines will not change.

[0041] Time when re-posting occurs: Re-posting mainly occurs within a period of time when a video is first released, but it may also occur at any time after a video is produced.

[0042] The impact of re-rotation on long and short videos: The impact on long videos is mainly that when the length of the video changes, the time point of the lines changes; when the size of the video changes, the relative position of the lines in the entire video changes. The impact on short videos is that short videos are derived from the interception of long videos. The corresponding time points of the short and long videos are retained during the interception process. Due to re-rotation, the corresponding relationship between the time points of the short and long videos has changed.

[0043] Using short videos alone for dialogue production will cause the following problems: if there are fewer lines in the short video, but text appears frequently in a non-dialogue position, the position will be mistakenly detected as a line and the real lines will be filtered out; or, if some other non-dialogue text falls into the position range corresponding to the lines, it will be mistakenly detected as a line.

[0044] The goal of dialogue production: Whether it is long video dialogue or short video dialogue, the main thing in the dialogue production process is to determine the text content, position and time point of each line in the video, where the time point includes the start time and end time.

[0045] The process of dialogue production: first sample the video frames at equal intervals, detect the areas with text in the sampled video frames, filter out the areas that do not correspond to the dialogue, and then track and identify the dialogue in the filtered areas to obtain the specific dialogue text content, location and time point.

[0046] Reference Figure 2 , shows a flowchart of a method for generating video lines in the present application, which may specifically include the following steps:

[0047] In step S11, the first line of the original video, the offset type of the second video compared to the original video, and the second line of the second video are obtained; the first video is obtained after re-transmitting the original video, and the second video is generated based on the video clip of the first video corresponding to the target time point.

[0048] In this step, the first speech of the original video, the offset type of the second video compared to the original video, and the second speech of the second video are first obtained. The content of the first speech and the second speech includes: speech text, speech position, speech time point and other information, and the speech time point includes the starting time point and the ending time point. The offset type includes time offset and position offset.

[0049] The line position refers to the position coordinates of the line subtitles in the video screen. In order for the audience to read the line text completely, each line needs to be displayed in the video screen for a certain period of time. This period of time is the time period corresponding to each line. The starting time point of the line refers to the starting time of the time period corresponding to the line, and the ending time point of the line refers to the ending time of the time period corresponding to the line.

[0050] The first video is obtained after re-transmitting the original video, that is, the re-transmitted video of the original video. The first video and the original video are usually long videos. If the length or screen content of the original video increases or decreases, the original video needs to be re-transmitted, and the first video is obtained after the re-transmission. Therefore, compared with the original video, some content of the first video will change. The re-transmission of the original video is mainly concentrated in a period of time when the original video is just online, but it may also occur at any time after the original video is generated.

[0051] After the original video is re-converted, the first video will have a slight change in video duration and video frames compared to the original video, resulting in a time offset for the second video compared to the original video. In some cases, there is also a change in video size, resulting in a position offset for the second video compared to the original video. For example, a few video frames may be deleted or added based on the original video, or the original video may have black borders above and below the video screen. By re-converting the video, the black borders above and below the video screen can be removed in the first video.

[0052] However, during the video re-conversion process, the main content of the first video will not change significantly compared to the original video, that is, the difference between the first video and the original video is within a small range. For example, if the main content of the original video is a plot of a movie, the main content of the first video will at least still be this movie, and will not change to another movie.

[0053] Therefore, the dialogue texts corresponding to the first video and the original video usually do not change, but the video size and video length may change. Usually, the change in video length will not exceed 2 minutes, which will lead to changes in the dialogue position and dialogue time point.

[0054] For example, in the original video, the position of the lines may be in a rectangular area 2 cm from the bottom of the video screen, while in the first video, the position of the lines changes to a rectangular area 1 cm from the bottom of the video screen; or, in the original video, the time point of a certain line of dialogue corresponds to 2:30 to 3:30, while in the first video, the time point of the same line of dialogue changes to 2:00 to 3:00.

[0055] The second video is generated based on the video clip of the first video corresponding to the target time point, which is usually a short video. In other words, the second video is generated based on the video clip intercepted from the first video. For example, if the target time point is 2:30 to 3:30, then the video clip from 2:30 to 3:30 can be firstly intercepted from the first video, and then part of the video frames can be further extracted from the intercepted video clip as the second video, or the intercepted video clip can be directly used as the second video, which is not limited in this application.

[0056] It is understandable that if the second video is used alone for dialogue production, the following problems will arise: if the second video has fewer lines, but text appears frequently in a non-dialogue position, the text appearing in that position will be mistakenly detected as the second line, while the text appearing in the actual dialogue position will be filtered out; or, if some other non-dialogue text falls into the dialogue position, it may also be mistakenly detected as the second video's lines, resulting in the target lines that are ultimately identified to be inaccurate.

[0057] Compared with the original video, the position and time of the lines in the first video may change, and the target time point corresponding to the first video is retained during the generation of the second video. Therefore, the line segment captured from the first line of the original video according to the target time point may not be completely matched with the second video. There is a certain time offset, and it is impossible to guide the production of the target lines.

[0058] In one implementation, obtaining the offset type of the second video compared to the original video includes:

[0059] Performing text detection on multiple target frames of the second video respectively, identifying regions where text exists, and obtaining multiple candidate text frames;

[0060] Detect whether the candidate text frame matches the position information of the first speech in the original video, and use the candidate text frame that matches the position information as the candidate speech frame;

[0061] Calculate the ratio between the number of candidate dialogue boxes and the number of candidate text boxes;

[0062] When the ratio is less than or equal to a second preset threshold, determining that the offset type of the second video compared to the original video is position offset;

[0063] When the ratio is greater than the second preset threshold, it is determined that the offset type of the second video compared to the original video is time offset.

[0064] That is to say, first, text detection can be performed on multiple target frames of the second video respectively, and areas with text in the second video can be identified, and these areas can be used as candidate text boxes, wherein the target frame of the second video can be every frame in the second video, or it can be a plurality of video frames obtained by sampling the second video according to a certain rule, or it can refer to a key frame in the second video, without specific limitation.

[0065] Furthermore, it is detected whether the candidate text box matches the position information of the first line in the original video. When a candidate text box matches the position information of the first line, it means that the candidate text box is located at the position where the line exists in the picture. Therefore, the candidate text box can be determined as a candidate line box. Conversely, when a candidate text box does not match the position information of the first line, it means that the candidate text box is not at the position where the line exists in the picture. Therefore, the candidate text box can be excluded.

[0066] Among them, matching the candidate text box with the location information may mean that the candidate text box completely falls within the area indicated by the location information, or it may mean that the intersection and union ratio between the candidate text box and the area indicated by the location information reaches a first preset threshold. This application does not make any specific limitations on this.

[0067] That is to say, for the second video, based on the first line position information pre-recorded in the database, it is possible to further determine from the candidate text boxes where text is detected that the included text may be part of the candidate text boxes for the lines, and then by calculating the ratio between the number of candidate line boxes and candidate text boxes, the offset type of the second video compared to the original video can be determined.

[0068] Specifically, the ratio of candidate dialogue boxes to candidate text boxes can be counted. When the ratio of candidate dialogue boxes to candidate text boxes is higher than a second preset threshold, it means that the candidate text boxes identified by text detection on the second video are most likely the area where the dialogue is located. It is understandable that most of the text in the video is indeed dialogue. In this case, it can be considered that after the original video is re-rotated, the position of the dialogue in the picture has not changed significantly. In other words, the video size has not been re-rotated. Therefore, the offset type is determined to be time offset.

[0069] If the ratio of the candidate dialogue box to the candidate text box is lower than a second preset threshold, it means that the candidate text box identified by text detection on the second video is less likely to be the area where the dialogue is located. In this case, it can be considered that after the original video is re-rotated, the position of the dialogue in the picture has changed significantly. In other words, the video size may be re-rotated. Therefore, the offset type is determined to be position offset.

[0070] The second preset threshold may be 40%, or may be other values, which are not specifically limited.

[0071] In one implementation, detecting whether the candidate text box matches the position information of the first speech in the original video, and using the candidate text box that matches the position information as the candidate speech box includes:

[0072] Calculate the intersection-and-union ratio between the candidate text box and the position information; the position information is the minimum bounding rectangle corresponding to the area where the first line is located;

[0073] A candidate text frame whose intersection-over-union ratio is greater than a first preset threshold is determined as a candidate dialogue frame.

[0074] In other words, the position information is the smallest rectangular frame that can contain all the areas where the first words are located, that is: count the position coordinates of the areas where all the first words are located, and determine the maximum value max_x and minimum value min_x of the horizontal coordinates and the maximum value max_y and minimum value min_y of the vertical coordinates of these position coordinates as the position information.

[0075] In this embodiment, whether the candidate text box matches the position information can be determined based on the intersection-over-union ratio between the two. The intersection-over-union ratio is the ratio of the intersection of the candidate text box and the position information to the union of the two, which can measure the overlap between the candidate text box and the position information.

[0076] That is to say, if the intersection ratio is greater than the first preset threshold, it means that the candidate text box has a high degree of overlap with the position information. In this case, the text included in the candidate text box is likely to be lines. Conversely, if the intersection ratio is less than or equal to the first preset threshold, it means that the candidate text box has a low degree of overlap with the position information. In this case, the text included in the candidate text box is unlikely to be lines.

[0077] Among them, the first preset threshold can be set according to the current application scenario and requirements, or it can be an empirical value based on existing experiments, and this application does not make any specific limitations on this.

[0078] In one implementation, obtaining the second lines of the second video includes:

[0079] When the offset type is time offset, text tracking is performed on the candidate speech frame to obtain the second speech of the second video.

[0080] The text detection on the second video and the text tracking on the candidate dialogue frame may be performed based on a pre-algorithm. For example, the pre-algorithm may include the following steps:

[0081] Extracting N video frames from the second video at a preset time interval, and performing text detection on the extracted video frames to obtain candidate text boxes, wherein the preset time interval may be 1 second or other values, which are not specifically limited, and N is an integer greater than 0;

[0082] When the candidate text box matches the position information, the candidate text box is used as a candidate dialogue box. Similarity comparison is performed based on the text features and text content corresponding to the candidate dialogue box in adjacent video frames to track the continuity of the dialogue. If the text features and text content corresponding to the candidate dialogue boxes in adjacent video frames are highly similar, it can be considered that these candidate dialogue boxes correspond to the same dialogue. Then, based on the timestamps of the adjacent video frames, the time point corresponding to the dialogue can be determined.

[0083] By continuously expanding to track the time points of the lines on each video frame, the starting time point and the ending time point of each line are expanded respectively, thereby realizing the judgment of the starting time point and the ending time point of the lines on the video frame level, and finally obtaining the second line of the second video.

[0084] In this way, some candidate lines frames whose included texts are not lines can be preliminarily eliminated to obtain second lines. The second lines have a certain accuracy rate and are of reference significance for the generation of target lines.

[0085] In one implementation, obtaining the second lines of the second video includes:

[0086] When the offset type is position offset, the candidate text boxes whose central point horizontal coordinates are outside the horizontal coordinate range of the position information are filtered to obtain the text box to be processed;

[0087] Generate a heat map of the text box to be processed, and update the position information based on the heat map, and use the candidate text box matching the updated position information as a new candidate dialogue box;

[0088] Text tracking is performed on the new candidate dialogue frame to obtain the second dialogue of the second video.

[0089] It is understandable that the position of the lines on the horizontal axis usually runs from the leftmost side to the rightmost side, and there will not be much difference. The difference mainly exists in the position of the lines on the vertical axis. For example, it may be 1 cm, 2 cm or 3 cm from the bottom of the screen. The height is affected by the font size, etc., and there is no specific limitation.

[0090] Therefore, in the case where there is a positional shift between the first video and the original video, the position of the lines can be repositioned by simply determining the size shift on the vertical axis.

[0091] Specifically, first determine the center point of each candidate text box, then filter out the candidate text boxes whose center point's horizontal coordinates are outside the horizontal coordinate range of min_x to max_x of the position information, and retain the candidate text boxes whose center point's horizontal coordinates are within the range of [min_x, max_x] as the text boxes to be processed.

[0092] Then, a heat map is generated for the text boxes to be processed that are retained after filtering, and the position information of the area where the lines are located is updated based on the heat map. Then, the intersection-and-union ratio between the candidate text boxes and the position information is recalculated, and the candidate lines boxes are re-determined to obtain the second lines of the second video.

[0093] Among them, the heat map can represent the distribution of the text box to be processed in the second video screen. If the heat map indicates that the heat of a certain area is higher, it means that the frequency of the text box to be processed appearing in the area is higher, and the area with a higher frequency of text appearance in the second video is usually the area where the lines are located. Therefore, it can be considered that the probability that the area is the location of the lines is higher. Conversely, if the heat map indicates that the heat of a certain area is lower, it means that the frequency of the text box to be processed appearing in the area is lower, and the probability that the area is the location of the lines is lower.

[0094] In this way, when the size of the video changes after being re-transferred, the position of the lines can also be corrected, thereby further improving the recognition accuracy of the target lines in the second video.

[0095] In one implementation, generating a heat map of a text box to be processed includes:

[0096] Create a two-dimensional array with the same screen size as the second video; the initial value of each data point in the two-dimensional array is 0;

[0097] Traverse the text boxes to be processed, add 1 to the data points corresponding to the positions of the text boxes to be processed in the second video in the two-dimensional array, and obtain a heat map;

[0098] That is to say, when generating a heat map of the text box to be processed, first create a two-dimensional array of the same size as the screen size of the second video, and the initial value of each data point is 0; then, for each detected text box to be processed, according to the position of the text box to be processed in the second video screen, determine its corresponding position in the two-dimensional array, and add 1 to the corresponding position value in the two-dimensional array.

[0099] Among them, there are multiple text boxes to be processed. Then, when all the text boxes to be processed are processed, the values ​​of each data point in the two-dimensional array are superimposed for multiple rounds to obtain a heat map. The larger the value of the data point in the heat map, the more times the text box to be processed appears at the corresponding position of the data point.

[0100] In one implementation, updating the location information based on the heat map includes:

[0101] Determine the highest value of the data point in the heat map;

[0102] The target value is obtained based on the highest value and the preset ratio;

[0103] Determine the data points with values ​​higher than the target value as the retained data points;

[0104] Determine the minimum bounding rectangle of the area corresponding to the retained data point as the updated position information.

[0105] That is to say, after obtaining the heat map, the highest value among the data points included in the heat map can be determined, and then the product of the highest value and the preset ratio can be calculated as the target value. Some data points with values ​​higher than the target value can be screened out from the data points included in the heat map as retained data points. For example, the preset ratio can be 50%, then all data points with values ​​higher than 50% of the highest value can be retained as retained data points.

[0106] Furthermore, the minimum enclosing rectangle of the area corresponding to the retained data point can be obtained. It can be understood that the value of each data point in the heat map is determined according to the frequency of the text box to be processed appearing in the position corresponding to the data point. The retained data point is the part with higher values ​​in the two-dimensional array. The higher the frequency of text appearing in the corresponding area, the higher the probability that the area is the location of the lines. Therefore, the obtained minimum enclosing rectangle can be used as the updated position information.

[0107] The number of the minimum bounding rectangles of the reserved area is related to the presentation format of the dialogue in the second video, which may be one, two, or more. For example, if there is one bounding rectangle, it means that the dialogue of the second video is only displayed one line at a time. If there are two bounding rectangles, and the positions of the two bounding rectangles are in an up-and-down relationship, it can be understood that the dialogue of the second video is displayed two lines at a time.

[0108] In this way, when the first video is resized relative to the original video, the position information of the lines in the second video can be updated to achieve repositioning of the lines, thereby facilitating subsequent correction of the lines in the second video.

[0109] In step S12, based on the target time point and the offset type, a reference line is extracted from the first line.

[0110] In this step, after obtaining the first line of the original video, the offset type of the second video compared to the original video, and the second line of the second video, different methods can be used to extract reference lines from the first line based on different offset types according to the target time point of the second video relative to the first video.

[0111] In one implementation, when the second video is offset from the original video, the reference line corresponding to the target time point can be directly intercepted from the first line based on the target time point. For example, if the target time point of a second line is 2 minutes and 0 seconds in the first video, the reference line corresponding to 2 minutes and 0 seconds can be intercepted from the first line.

[0112] In another implementation, when the second video is time-shifted compared to the original video, based on the target time point and the type of the shift, extracting reference lines from the first lines includes:

[0113] Based on the target time point and the preset offset range, determine the time point corresponding to the second line in the original video as a reference time range;

[0114] A reference line corresponding to a reference time range is extracted from the first line.

[0115] That is to say, according to the target time point of the second video relative to the first video and the pre-set preset offset range, the reference time range of the lines of the second video relative to the original video is determined. For example, if the preset offset range is 2 minutes, then the reference lines can be extracted from the first lines by judging the reference time range of [-2 minutes, +2 minutes] before and after the corresponding target time point of the second lines one by one.

[0116] For example, if the target time point of a second line of dialogue is 2 minutes and 0 seconds in the first video, the reference time range of this line relative to the original video can be determined to be 0 minutes and 0 seconds to 4 minutes and 0 seconds based on the preset offset range. Then, the reference lines corresponding to 0 minutes and 0 seconds to 4 minutes and 0 seconds can be extracted from the first line.

[0117] In this way, the portion of the first line that may correspond to each second line can be quickly determined, and the reference lines can be used to guide the production of the target lines, thereby facilitating the correction of the second lines in subsequent steps.

[0118] In step S13, the second lines are filtered according to the reference lines and the offset type to obtain the target lines of the second video.

[0119] In this step, the second lines can be filtered in different ways based on the reference lines and different offset types, and the content in the second lines that does not exist in the reference lines can be filtered out to obtain the target lines of the second video, thereby correcting the second lines. In this way, the production of the target lines for the second video is guided by the reference lines of the original video, which can reduce the impact of the offset between the second video and the original video, and finally obtain more accurate target lines that match the second video.

[0120] In one implementation, if the second video is offset from the original video, the second lines are filtered according to the reference lines and the offset type to obtain the target lines of the second video, including:

[0121] Taking each line of the second line as the text to be filtered in turn, and determining the first reference time point corresponding to the text to be filtered;

[0122] Detect whether the speech text corresponding to the first reference time point in the first speech matches the text to be filtered;

[0123] When the speech text corresponding to the first reference time point in the first speech does not match the text to be filtered, the text to be filtered is deleted from the second speech to obtain the target speech of the second video.

[0124] That is to say, by detecting whether the speech text corresponding to the first reference time point in the first speech matches the text to be filtered one by one, the unmatched speech text is filtered out. Among them, whether the speech text matches the text to be filtered can be achieved by comparing the speech text with the text to be filtered, that is, if the text content of the two is the same, the match is successful, otherwise the match fails; or, it can also be achieved by comparing the character length of the speech text with the text to be filtered, that is, if the character length of the two is the same, the match is successful, otherwise the match fails; and so on, there is no specific limitation.

[0125] In another implementation, if the second video is time-shifted compared to the original video, the second lines are filtered according to the reference lines and the type of the shift to obtain the target lines of the second video, including:

[0126] Calculating the similarity between the second speech and the reference speech in the time dimension, and determining the offset time between the first speech and the second speech based on the similarity;

[0127] The second lines are filtered according to the offset time and the first lines to obtain the target lines of the second video.

[0128] In this embodiment, the similarity between the second lines and the reference lines in the time dimension can be calculated, that is, the similarity between the second lines and the reference lines at different times. It can be understood that the line texts of the second lines and the reference lines have a certain consistency, that is, the line text of a certain sentence in the second lines is likely to be the same as a certain sentence in the first lines, and have a high similarity. Therefore, through the similarity between the second lines and the reference lines, the second lines and the first lines with higher line text similarity can be determined, and then the offset time between the first line and the second line can be determined.

[0129] Among them, after determining the similarity between the second lines and the reference lines in the time dimension, the reference lines with the highest similarity corresponding to each second line can be further determined, and then, for each second line, the difference between the time point of the second line and the time point of its corresponding reference line is calculated as the time offset between the two, and then, the average value of the time offsets corresponding to each second line is calculated as the offset time between the first line and the second line.

[0130] Specifically, if the target time point corresponding to a certain line text in the second line is A, and the time point corresponding to the first line with a higher similarity to it in the original video is A', the target time point of another second line in the second video is B, and the time point corresponding to the first line with a higher similarity to it in the original video is B', and so on, the second lines with target time points ABCDEFG are continuously matched, among which the second line with target time point C may not match the corresponding first line, but the second line with target time point ABDEFG matches the first lines with time points A'B'D'E'F'G' respectively, then by calculating the average of the time offsets between A and A', B and B', ..., G and G', the offset time between the first line and the second line can be calculated.

[0131] Then, the first line corresponding to each second line can be determined based on the offset time and the first line. It can be understood that, usually, the second video is captured from the first video, and the first video is re-converted from the original video. Therefore, each line of the second video should exist in the first line of the original video. The second line that does not exist in the first line may be mistakenly identified as a line when the second video is detected for text. Then, based on the time point and offset time of the second line, the first line corresponding to the second line after eliminating the time offset can be determined, and the second line can be filtered based on the determined first line. The second line that does not exist in the first line is deleted, and the target line of the second video can be obtained, thereby realizing the correction of the lines of the second video.

[0132] In one implementation, filtering the second speech according to the offset time and the first speech to obtain the target speech of the second video includes:

[0133] Taking each line of the second line as the text to be filtered in turn, and determining the second reference time point corresponding to the text to be filtered according to the offset time and the target time point;

[0134] Detecting whether the speech text corresponding to the second reference time point in the first speech matches the text to be filtered;

[0135] When the speech text corresponding to the second reference time point in the first speech does not match the text to be filtered, the text to be filtered is deleted from the second speech to obtain the target speech of the second video.

[0136] That is to say, after obtaining the offset time, the unmatched speech texts can be filtered out by detecting whether the speech text corresponding to the second reference time point in the first speech matches the text to be filtered one by one based on the offset time. Among them, whether the speech text matches the text to be filtered can be achieved by comparing the speech text with the text to be filtered, that is, if the text content of the two is the same, the match is successful, otherwise the match fails; or, it can also be achieved by comparing the character length of the speech text with the text to be filtered, that is, if the character length of the two is the same, the match is successful, otherwise the match fails; and so on, there is no specific limitation.

[0137] For example, if the target time point corresponding to a sentence of text to be filtered is 2 minutes and 30 seconds, and the offset time is +20 seconds, then the first line with a time point of 2 minutes and 50 seconds can be obtained, and the text to be filtered is matched with the line text of the first line. If the two are the same, the text to be filtered is retained. If the two are different, it means that the text to be filtered does not exist in the first line, that is, the text to be filtered is likely to be a non-line text that was mistakenly identified. Therefore, the text to be filtered can be deleted from the second line to obtain the target line of the second video.

[0138] Reference Figure 3 , shows a flow chart of another method for generating video lines in the present application, which may specifically include:

[0139] In step S21, the first line of the original video is obtained.

[0140] In step S22, text detection is performed on multiple target frames of the second video respectively, and areas with text are identified to obtain multiple candidate text boxes; the first video is obtained after re-converting the original video, and the second video is generated based on the video clip of the first video corresponding to the target time point.

[0141] In step S23, the intersection-and-union ratio between the candidate text box and the position information is calculated; the position information is the minimum circumscribed rectangular box corresponding to the area where the first speech is located; and the candidate text box whose intersection-and-union ratio is greater than a first preset threshold is determined as the candidate speech box.

[0142] In step S24, the ratio between the number of candidate dialogue boxes and the number of candidate text boxes is calculated. If the ratio is greater than a second preset threshold, step S25 is executed; if the ratio is less than or equal to the second preset threshold, step S29 is executed.

[0143] In step S25, text tracking is performed on the candidate speech frame to obtain the second speech of the second video.

[0144] In step S26, based on the target time point and the preset offset range, the time point corresponding to the second speech in the original video is determined as a reference time range; and the reference speech corresponding to the reference time range is intercepted from the first speech.

[0145] In step S27 , the similarity between the second speech and the reference speech in the time dimension is calculated, and the offset time between the first speech and the second speech is determined based on the similarity.

[0146] In step S28, each line of the second speech is used as the text to be filtered in turn, and the second reference time point corresponding to the text to be filtered is determined according to the offset time and the target time point; it is detected whether the speech text corresponding to the second reference time point in the first speech matches the text to be filtered; if the speech text corresponding to the second reference time point in the first speech does not match the text to be filtered, the text to be filtered is deleted from the second speech to obtain the target speech of the second video.

[0147] In step S29, candidate text boxes whose central point abscissas are outside the abscissa range of the position information are filtered to obtain text boxes to be processed.

[0148] In step S30, a two-dimensional array with the same screen size as the second video is created; the initial value of each data point in the two-dimensional array is 0; the text boxes to be processed are traversed, and the values ​​of the data points corresponding to the positions of the text boxes to be processed in the two-dimensional array in the second video are added by 1 to obtain a heat map.

[0149] In step S31, the highest value of the data points in the heat map is determined; the target value is obtained based on the maximum value and the preset ratio; the data points with values ​​higher than the target value are determined as the retained data points; the minimum circumscribed rectangular box of the area corresponding to the retained data points is determined as the updated position information, and the candidate text box matching the updated position information is used as the new candidate dialogue box; the new candidate dialogue box is text tracked to obtain the second dialogue of the second video, and the process returns to step S25.

[0150] The specific implementation method of each step in this embodiment is as follows: Figure 1 The same is true in the illustrated embodiment and will not be described again here.

[0151] As can be seen from the above, the technical solution provided by the embodiment of the present disclosure, when the duration of the first video after the original video is re-converted changes compared to the original video, resulting in a time point offset of the second video generated based on the first video relative to the original video, when producing the target lines of the second video, first, according to the target time point corresponding to the second video and the offset type of the second video compared to the original video, the reference lines are intercepted from the first lines produced based on the original video, then, the second lines are corrected based on the reference lines and the offset type, and the production of the target lines for the second video is guided by the reference lines of the original video, thereby eliminating the offset between the second video and the original video, and finally obtaining a more accurate target line that matches the second video, thereby improving the recognition accuracy of the target lines and facilitating subsequent query and use.

[0152] Reference Figure 4 , shows a schematic diagram of the structure of a video dialogue generation device of the present application, which may specifically include:

[0153] The acquisition module 301 is used to acquire the first line of the original video, the offset type of the second video compared to the original video, and the second line of the second video; the first video is obtained after re-transmitting the original video, and the second video is generated based on the video segment of the first video corresponding to the target time point;

[0154] An interception module 302, configured to intercept reference lines from the first lines based on the target time point and the offset type;

[0155] The filtering module 304 is used to filter the second lines according to the reference lines and the offset type to obtain target lines of the second video.

[0156] Optionally, the acquisition module 301 is specifically configured to:

[0157] Performing text detection on multiple target frames of the second video respectively, identifying regions where text exists, and obtaining multiple candidate text frames;

[0158] Detecting whether the candidate text frame matches the position information of the first speech in the original video, and using the candidate text frame that matches the position information as a candidate speech frame;

[0159] Calculating a ratio between the number of the candidate dialogue frames and the number of the candidate text frames;

[0160] When the ratio is less than or equal to the second preset threshold, determining that the offset type of the second video compared to the original video is position offset;

[0161] When the ratio is greater than the second preset threshold, it is determined that the offset type of the second video compared to the original video is time offset.

[0162] Optionally, the acquisition module 301 is specifically configured to:

[0163] Calculating an intersection-and-union ratio between the candidate text box and the position information; the position information is a minimum bounding rectangle corresponding to the area where the first line is located;

[0164] The candidate text frame whose intersection-over-union ratio is greater than a first preset threshold is determined as a candidate dialogue frame.

[0165] Optionally, the acquisition module 301 is specifically configured to:

[0166] When the offset type is time offset, text tracking is performed on the candidate speech frame to obtain the second speech of the second video.

[0167] Optionally, the acquisition module 301 is specifically configured to:

[0168] In the case where the offset type is position offset, the candidate text boxes whose central point horizontal coordinates are outside the horizontal coordinate range of the position information are filtered to obtain a text box to be processed;

[0169] Generate a heat map of the text box to be processed, and update the position information based on the heat map, and use the candidate text box that matches the updated position information as a new candidate dialogue box;

[0170] Text tracking is performed on the new candidate dialogue frame to obtain the second dialogue of the second video.

[0171] Optionally, the acquisition module 301 is specifically configured to:

[0172] Creating a two-dimensional array having the same screen size as the second video; the initial value of each data point in the two-dimensional array is 0;

[0173] The text box to be processed is traversed, and 1 is added to the value of the data point corresponding to the position of the text box to be processed in the second video in the two-dimensional array to obtain a heat map.

[0174] Optionally, the acquisition module 301 is specifically configured to:

[0175] Determining the highest value of the data points in the heat map;

[0176] The target value is obtained based on the highest value and the preset ratio;

[0177] Determine the data points with values ​​higher than the target value as the retained data points;

[0178] The minimum bounding rectangle of the area corresponding to the retained data point is determined as the updated position information.

[0179] Optionally, the interception module 302 is specifically configured to:

[0180] When the offset type is position offset, a reference line corresponding to the target time point is extracted from the first line.

[0181] Optionally, the filtering module 304 is specifically configured to:

[0182] When the offset type is position offset, each line text in the second line is sequentially used as a text to be filtered, and a first reference time point corresponding to the text to be filtered is determined;

[0183] Detecting whether the speech text corresponding to the first reference time point in the first speech matches the text to be filtered;

[0184] When the speech text corresponding to the first reference time point in the first speech does not match the text to be filtered, the text to be filtered is deleted from the second speech to obtain the target speech of the second video.

[0185] Optionally, the interception module 302 is specifically configured to:

[0186] In the case where the offset type is time offset, based on the target time point and the preset offset range, determining the time point corresponding to the second line in the original video as a reference time range;

[0187] A reference line corresponding to a reference time range is extracted from the first line.

[0188] Optionally, the filtering module 304 is specifically configured to:

[0189] When the offset type is a time offset, calculating the similarity between the second speech and the reference speech in a time dimension, and determining the offset time between the first speech and the second speech based on the similarity;

[0190] The second lines are filtered according to the offset time and the first lines to obtain target lines of the second video.

[0191] Optionally, the filtering module 304 is specifically configured to:

[0192] Taking each line text in the second line as the text to be filtered in turn, and determining the second reference time point corresponding to the text to be filtered according to the offset time and the target time point;

[0193] Detecting whether the speech text corresponding to the second reference time point in the first speech matches the text to be filtered;

[0194] When the speech text corresponding to the second reference time point in the first speech does not match the text to be filtered, the text to be filtered is deleted from the second speech to obtain the target speech of the second video.

[0195] As can be seen from the above, the technical solution provided by the embodiment of the present disclosure, when the duration of the first video after the original video is re-converted changes compared to the original video, resulting in a time point offset of the second video generated based on the first video relative to the original video, when producing the target lines of the second video, first, according to the target time point corresponding to the second video and the offset type of the second video compared to the original video, the reference lines are intercepted from the first lines produced based on the original video, then, the second lines are corrected based on the reference lines and the offset type, and the production of the target lines for the second video is guided by the reference lines of the original video, thereby eliminating the offset between the second video and the original video, and finally obtaining a more accurate target line that matches the second video, thereby improving the recognition accuracy of the target lines and facilitating subsequent query and use.

[0196] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0197] Figure 5 The present invention is a block diagram of an electronic device for generating video dialogues according to an exemplary embodiment, which includes a processor and a memory, wherein the memory is used to store computer programs; and the processor is used to execute the programs stored in the memory.

[0198] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0199] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0200] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of an electronic device to perform the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0201] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed on a computer, the computer implements the above-mentioned method for generating commentary video lines.

[0202] As can be seen from the above, the technical solution provided by the embodiment of the embodiment of the present invention, when the duration of the first video after the original video is re-converted changes compared to the original video, resulting in a time point offset of the second video generated based on the first video relative to the original video, when producing the target lines of the second video, first, according to the target time point corresponding to the second video and the offset type of the second video compared to the original video, the reference lines are intercepted from the first lines produced based on the original video, then, the second lines are corrected based on the reference lines and the offset type, and the production of the target lines for the second video is guided by the reference lines of the original video, thereby eliminating the offset between the second video and the original video, and finally obtaining a more accurate target line that matches the second video, thereby improving the recognition accuracy of the target lines and facilitating subsequent query and use.

[0203] Figure 6 It is a block diagram of a device 800 for generating video dialogue according to an exemplary embodiment.

[0204] For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast electronic device, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0205] Reference Figure 6, the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0206] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0207] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0208] The power supply component 807 provides power to the various components of the device 800. The power supply component 807 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0209] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the account. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the account. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data to be processed. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0210] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0211] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0212] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0213] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0214] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the methods described in the first and second aspects.

[0215] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to complete the above method. Optionally, for example, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0216] In an exemplary embodiment, a computer program product including instructions is also provided. When the computer program product is run on a computer, the computer is enabled to execute the video dialogue generation method described in any one of the above embodiments.

[0217] As can be seen from the above, the technical solution provided by the embodiment of the embodiment of the present invention is that when the original video is re-transmitted, the duration of the obtained first video changes compared to the original video, resulting in that the second video will also be offset relative to the time period corresponding to the first video. In this case, when producing the target lines of the second video, first, according to the target time point corresponding to the second video and the offset type of the second video compared to the original video, the reference lines are intercepted from the first lines produced based on the original video, and then, the second lines are corrected based on the reference lines and the offset type, and the reference lines of the original video are used to guide the production of the target lines for the second video, thereby eliminating the offset between the second video and the original video, and obtaining a more accurate target line that matches the second video, thereby improving the recognition accuracy of the target lines and facilitating subsequent query and use.

[0218] Those skilled in the art will readily appreciate other implementations of the embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The embodiments of the present invention are intended to cover any variations, uses, or adaptations of the embodiments of the present invention that follow the general principles of the embodiments of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the embodiments of the present invention. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the embodiments of the present invention are indicated by the following claims.

[0219] It should be understood that the embodiments of the present invention are not limited to the precise structures described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the embodiments of the present invention is limited only by the appended claims.

Claims

1. A video dialogue generation method, characterized in that: include: Obtaining a first line of an original video, an offset type of a second video compared to the original video, and a second line of the second video; The first video is obtained by re-converting the original video, and the second video is generated based on a video segment of the first video corresponding to a target time point; Based on the target time point and the offset type, extracting a reference line from the first line; The second lines are filtered according to the reference lines and the offset type to obtain target lines of the second video.

2. The method according to claim 1, characterized in that The obtaining of the offset type of the second video compared to the original video includes: Performing text detection on multiple target frames of the second video respectively, identifying regions where text exists, and obtaining multiple candidate text frames; Detecting whether the candidate text frame matches the position information of the first speech in the original video, and using the candidate text frame that matches the position information as a candidate speech frame; Calculating a ratio between the number of the candidate dialogue frames and the number of the candidate text frames; When the ratio is less than or equal to the second preset threshold, determining that the offset type of the second video compared to the original video is position offset; When the ratio is greater than the second preset threshold, it is determined that the offset type of the second video compared to the original video is time offset.

3. The method according to claim 2, characterized in that The detecting whether the position information of the candidate text frame matches the first speech in the original video, and taking the candidate text frame matching the position information as a candidate speech frame, includes: Calculating an intersection-and-union ratio between the candidate text box and the position information; the position information is a minimum circumscribed rectangular box of an area where the first line is located; The candidate text frame whose intersection-over-union ratio is greater than a first preset threshold is determined as a candidate dialogue frame.

4. The method according to claim 2, characterized in that: The obtaining the second lines of the second video includes: When the offset type is time offset, text tracking is performed on the candidate speech frame to obtain the second speech of the second video.

5. The method according to claim 2, characterized in that: The obtaining the second lines of the second video includes: In the case where the offset type is position offset, the candidate text boxes whose central point horizontal coordinates are outside the horizontal coordinate range of the position information are filtered to obtain a text box to be processed; Generate a heat map of the text box to be processed, and update the position information based on the heat map, and use the candidate text box that matches the updated position information as a new candidate dialogue box; Text tracking is performed on the new candidate dialogue frame to obtain the second dialogue of the second video.

6. The method according to claim 5, characterized in that The step of generating a heat map of the text box to be processed includes: Creating a two-dimensional array having the same screen size as the second video; the initial value of each data point in the two-dimensional array is 0; The text box to be processed is traversed, and 1 is added to the value of the data point corresponding to the position of the text box to be processed in the second video in the two-dimensional array to obtain a heat map.

7. The method according to claim 5, characterized in that The updating of the location information based on the heat map includes: Determining the highest value of the data points in the heat map; The target value is obtained based on the highest value and the preset ratio; Determine the data points with values ​​higher than the target value as the retained data points; The minimum bounding rectangle of the area corresponding to the retained data point is determined as the updated position information.

8. The method according to claim 1, characterized in that The extracting a reference line from the first line based on the target time point and the offset type includes: When the offset type is position offset, a reference line corresponding to the target time point is extracted from the first line.

9. The method according to claim 1, characterized in that: The filtering the second lines according to the reference lines and the offset type to obtain target lines of the second video includes: When the offset type is position offset, each line text in the second line is sequentially used as a text to be filtered, and a first reference time point corresponding to the text to be filtered is determined; Detecting whether the speech text corresponding to the first reference time point in the first speech matches the text to be filtered; When the speech text corresponding to the first reference time point in the first speech does not match the text to be filtered, the text to be filtered is deleted from the second speech to obtain the target speech of the second video.

10. The method according to claim 1, characterized in that The extracting a reference line from the first line based on the target time point and the offset type includes: When the offset type is time offset, based on the target time point and the preset offset range, determining the time point corresponding to the second line in the original video as a reference time range; A reference line corresponding to the reference time range is extracted from the first line.

11. The method according to claim 1, characterized in that: The filtering the second lines according to the reference lines and the offset type to obtain target lines of the second video includes: When the offset type is a time offset, calculating the similarity between the second speech and the reference speech in the time dimension, and determining the offset time between the first speech and the second speech based on the similarity; The second lines are filtered according to the offset time and the first lines to obtain target lines of the second video.

12. The method according to claim 11, characterized in that The filtering the second lines according to the offset time and the first lines to obtain target lines of the second video includes: Taking each line text in the second line as the text to be filtered in turn, and determining the second reference time point corresponding to the text to be filtered according to the offset time and the target time point; Detecting whether the speech text corresponding to the second reference time point in the first speech matches the text to be filtered; When the speech text corresponding to the second reference time point in the first speech does not match the text to be filtered, the text to be filtered is deleted from the second speech to obtain the target speech of the second video.

13. A video dialogue generation device, characterized in that: include: An acquisition module, configured to acquire a first line of an original video, an offset type of a second video compared to the original video, and a second line of the second video; The first video is obtained by re-converting the original video, and the second video is generated based on a video segment of the first video corresponding to a target time point; An interception module, configured to intercept reference lines from the first lines based on the target time point and the offset type; A filtering module is used to filter the second lines according to the reference lines and the offset type to obtain target lines of the second video.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the video dialogue generation method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video dialogue generation method according to any one of claims 1 to 12 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the video dialogue generation method according to any one of claims 1 to 12 is implemented.

Images