Picture generation method and device, electronic equipment and storage medium

Abstract

The invention relates to a picture generation method and device, electronic equipment and a storage medium. The method comprises the following steps: extracting a plurality of video frames in a target video; determining target boundary information of a main picture area in the target video, wherein the target boundary information comprises a target upper boundary position and a target lower boundary position of the main picture area; cutting each video frame in the plurality of video frames according to the target boundary information to obtain a plurality of main body area maps; determining at least one line set corresponding to at least one role in the target video; according to the at least one line set and the main body area graph corresponding to each line in each line set, generating at least one spliced picture, the single spliced picture comprises a main picture region in the main region picture corresponding to the first line in the single line set and line regions in the main region picture corresponding to other lines except the first line. Therefore, high-quality spliced pictures can be automatically produced in batches with low cost and high efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a method and device for generating an image, an electronic device, and a storage medium. Background Art

[0002] In the film and television industry, film and television publicity and promotion is a critical link, and can even determine the market performance of a work to a certain extent. Its importance is reflected in many aspects: carefully planned publicity and promotion strategies in the early stage can create popularity for the work in advance and stimulate the audience's curiosity and expectations; attracting the audience's attention through trailers, posters, stills and other forms of promotion can effectively increase the public awareness of film and television works; through accurate positioning of content and market segmentation during the publicity and promotion process, and using popular short video platforms and social media platforms for promotion, it can help the work reach the target audience more effectively.

[0003] Currently, promotional materials for film and television content on social media platforms are usually produced manually (such as spliced ​​pictures of lines, spliced ​​pictures of characters, etc.). In particular, pictures are usually positively correlated with the number of readings and interactions of the published content, but the manual production method is costly and inefficient. Summary of the invention

[0004] In view of this, the present disclosure proposes a method and device for generating an image, an electronic device and a storage medium, which can automatically batch-produce high-quality spliced ​​images at low cost and high efficiency.

[0005] According to one aspect of the present disclosure, a method for generating an image is provided, comprising: acquiring a target video and extracting a plurality of video frames in the target video; determining target boundary information of a main screen area in the target video based on the plurality of video frames, the target boundary information comprising a target upper boundary position and a target lower boundary position of the main screen area; cropping each of the plurality of video frames according to the target boundary information to obtain a plurality of main screen area maps, the main screen area maps comprising the main screen area in the video frames; determining at least one line set corresponding to at least one character in the target video; wherein a single line set comprises a plurality of lines of contextual association of a single character and a display position of each line, and the plurality of lines in the single line set are arranged in a frame order of the video frames to which the main screen area map belongs; generating at least one spliced ​​image according to the at least one line set and the main screen area map corresponding to each line in each line set, the single spliced ​​image comprising a main screen area in the main screen area map corresponding to a first line in the single line set and a line area in the main screen area map corresponding to the remaining lines except the first line.

[0006] In a possible implementation, the target boundary information of the main picture area in the target video is determined based on the multiple video frames, including: for any video frame in the multiple video frames, by comparing the pixel value differences between adjacent pixels in each column in the video frame, the upper boundary position of each column includes the pixel position where the pixel value first mutates when the pixel value differences are compared in a top-to-bottom order in each column, and the lower boundary position of each column includes the pixel position where the pixel value first mutates when the pixel value differences are compared in a bottom-to-top order in each column; according to the upper boundary position and the lower boundary position of each column in the video frame, the boundary information corresponding to the video frame is determined, the boundary information including the regional upper boundary position and the regional lower boundary position of the main picture area of ​​the video frame; according to the boundary information corresponding to each video frame in the multiple video frames, the target boundary information is determined.

[0007] In a possible implementation, the obtaining of the upper boundary position and the lower boundary position of each column by comparing the pixel value differences of adjacent pixels in each column of the video frame includes: for the i-th column in the video frame, comparing the pixel value differences between adjacent pixels in the i-th column in the video frame in a top-to-bottom order and a bottom-to-top order, respectively, to obtain the first pixel position where the pixel value of the i-th column first undergoes a sudden change when the pixel value differences are compared in a top-to-bottom order and the second pixel position where the pixel value first undergoes a sudden change when the pixel value differences are compared in a bottom-to-top order; starting from a row of pixels at the first pixel position, Whether the pixel values ​​of the pixels in the same row are continuously the same when traversing upward row by row from the first pixel position, and whether the pixel values ​​of the pixels in the same row are continuously the same when traversing downward row by row from the second pixel position, obtain the third pixel position when the pixel values ​​of the pixels in the same row are continuously the same for the first time when traversing upward from the first pixel position, and the fourth pixel position when the pixel values ​​of the pixels in the same row are continuously the same for the first time when traversing downward from the second pixel position; determine the third pixel position as the upper boundary position of the i-th column, and determine the fourth pixel position as the lower boundary position of the i-th column.

[0008] In a possible implementation, the pixel positions of the video frame from top to bottom are in ascending order, and the boundary information corresponding to the video frame is determined based on the upper boundary position and the lower boundary position of each column in the video frame, including: determining the maximum value of multiple upper boundary positions corresponding to multiple columns in the video frame as the regional upper boundary position of the main picture area of ​​the video frame; determining the minimum value of multiple lower boundary positions corresponding to multiple columns in the video frame as the regional lower boundary position of the main picture area of ​​the video frame.

[0009] In a possible implementation, the target boundary information is determined based on the boundary information corresponding to each video frame in the multiple video frames, including: filtering outlier boundary information in the multiple boundary information corresponding to the multiple video frames to obtain multiple valid boundary information in the multiple boundary information; determining the average value of the upper boundary positions of the region in the multiple valid boundary information as the target upper boundary position of the main picture region in the target video; and determining the average value of the lower boundary positions of the region in the multiple valid boundary information as the target lower boundary position of the main picture region in the target video.

[0010] In one possible implementation, determining at least one line set corresponding to at least one character in the target video includes: extracting line data in each of the multiple main area maps, the line data including the lines in the main area map and the display position of the lines; constructing a global line set based on the line data in each of the multiple main area maps, the global line set including all lines in the multiple main area maps arranged in the frame order of the video frames to which the main area maps belong; and determining at least one line set corresponding to each of at least one character according to the global line set using an artificial intelligence model.

[0011] In a possible implementation, generating at least one spliced ​​image based on the at least one line set and the main area map corresponding to each line in each line set includes: for any line set, according to the display position of at least one remaining line in the line set except the first line, cutting out the line area map of each remaining line in the main area map to which they belong; and splicing the main area map corresponding to the first line with the line area maps corresponding to the remaining lines in accordance with the frame order of the video frames to which the main area map corresponding to each line in the line set belongs, to obtain a spliced ​​image.

[0012] In a possible implementation, the method further includes: identifying a facial area in each of the multiple subject area images and evaluating the clarity of the facial area to obtain a facial clarity score for each subject area image; screening out at least two target subject area images from the multiple subject area images based on the facial clarity score of each subject area image; and horizontally or vertically splicing the at least two target subject areas based on the aspect ratio of the subject area images to obtain a spliced ​​image, wherein the aspect ratio is the ratio between the width and the height of the subject area image.

[0013] In a possible implementation, when the target upper boundary position is different from the top boundary position of the video frame, the method further includes: for any video frame, according to the target upper boundary position, cropping the title area in the video frame to obtain a title area map of the video frame; performing text recognition on the title area map to obtain a video title of the video frame; the video title is used to assemble promotional materials on the social platform with the spliced ​​picture.

[0014] According to another aspect of the present disclosure, a picture generation device is provided, comprising: an extraction module, configured to obtain a target video and extract a plurality of video frames in the target video; a boundary determination module, configured to determine target boundary information of a main screen area in the target video based on the plurality of video frames, wherein the target boundary information includes a target upper boundary position and a target lower boundary position of the main screen area; a cropping module, configured to crop each of the plurality of video frames according to the target boundary information to obtain a plurality of main screen area maps, wherein the main screen area maps include the main screen area in the video frame; and a line determination module, configured to determine the target At least one dialogue set corresponding to at least one character in a video; wherein a single dialogue set includes multiple dialogues associated with a single character in context and the display position of each dialogue, and the multiple dialogues in the single dialogue set are arranged in the frame order of the video frames to which the main area map belongs; an image generation module, for generating at least one spliced ​​image according to the at least one dialogue set and the main area map corresponding to each line in each dialogue set, wherein the single spliced ​​image includes a main screen area in the main area map corresponding to a first line in the single dialogue set and a dialogue area in the main area map corresponding to the remaining lines except the first line.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0016] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0018] According to various aspects of the present disclosure, by extracting multiple video frames of a target video, identifying the target upper boundary position and the target lower boundary of the main screen area, cropping a main area map containing the main screen area in the video frame, and then determining at least one line set corresponding to at least one character in the target video, and generating at least one spliced ​​image according to the main screen area in the main area map corresponding to the first line of the lines in the line set and the line areas in the main area map corresponding to the remaining lines except the first line, it is possible to automatically generate context-related line splicing images divided by roles based on any target video, and realize low-cost and high-efficiency automatic batch production of high-quality spliced ​​images. In an exemplary scenario, the spliced ​​images can be used as promotional materials for film and television dramas on social platforms.

[0019] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0021] Figure 1 A flowchart of a method for generating an image according to an embodiment of the present disclosure is shown.

[0022] Figure 2 A schematic diagram of a frame image of a target video according to an embodiment of the present disclosure is shown.

[0023] Figure 3 A schematic diagram of another frame image of a target video according to an embodiment of the present disclosure is shown.

[0024] Figure 4 A schematic diagram showing target boundary information according to an embodiment of the present disclosure.

[0025] Figure 5 A schematic diagram showing an exemplary video frame according to an embodiment of the present disclosure.

[0026] Figure 6a A schematic diagram of a main area map according to an embodiment of the present disclosure is shown.

[0027] Figure 6b A schematic diagram showing another main area diagram according to an embodiment of the present disclosure.

[0028] Figure 7 A schematic diagram showing a process for determining a dialogue set according to an embodiment of the present disclosure is shown.

[0029] Figure 8a and Figure 8bA schematic diagram showing two stitched pictures according to an embodiment of the present disclosure is shown.

[0030] Figure 9a A schematic diagram showing a stitched picture obtained by horizontal stitching according to an embodiment of the present disclosure is shown.

[0031] Figure 9b A schematic diagram showing a stitched picture obtained by vertical stitching according to an embodiment of the present disclosure is shown.

[0032] Fig.10 A schematic diagram showing a picture generation process according to an embodiment of the present disclosure.

[0033] Fig.11 A block diagram of a picture generating device according to an embodiment of the present disclosure is shown.

[0034] Fig.12 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0036] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0037] The term "and / or" herein is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent any one or more elements selected from the set consisting of A, B, and C. In the description of the present disclosure, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0038] It should be understood that the terms "first", "second", etc. in the claims, specifications and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0039] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0040] The image generation method of the embodiment of the present disclosure can be deployed on various terminal devices through software or hardware modification. The terminal device involved in the embodiment of the present disclosure may refer to a device with a wireless connection function and / or a wired connection function. The wireless connection function refers to the ability to connect to other devices through wireless connection methods such as wifi and Bluetooth. The terminal device involved in the embodiment of the present disclosure may also communicate with other devices through a wired connection function. The terminal device involved in the embodiment of the present disclosure may be a touch screen, a non-touch screen, or a screenless terminal device. The touch screen terminal device can be controlled by clicking and sliding on the display screen with a finger or a stylus. The non-touch screen device can be connected to an input device such as a mouse, a keyboard, a touch panel, and the terminal device is controlled through the input device. For example, the device without a screen may be a Bluetooth speaker without a screen. For example, the terminal device of the present application may include but is not limited to a user equipment (UE), a mobile device, a user terminal, a terminal, a handheld device, a tablet computer, a laptop, a PDA, a computing device, etc.

[0041] The image generation method of the embodiment of the present disclosure can also be deployed on a server, which can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine, a container, etc., with a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. It can refer to a device with a wireless connection function, and the wireless connection function refers to the ability to connect to other servers or terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The server involved in the embodiment of the present disclosure may also have the function of communicating through a wired connection. For example, the server of the embodiment of the present disclosure can be located in the cloud, communicate with the terminal device, receive the target video sent by the terminal device, and use the image generation method deployed on the server to generate at least one spliced ​​picture based on the target video, and return it to the terminal device to display the generated at least one spliced ​​picture to the user in the terminal device.

[0042] Figure 1 FIG. 1 is a flow chart showing a method for generating an image according to an embodiment of the present disclosure. The method can be applied to the above-mentioned terminal device or server. Figure 1 As shown, the method includes: step S11 to step S15.

[0043] In step S11, a target video is acquired, and a plurality of video frames in the target video are extracted.

[0044] Considering that most of the pictures used for film and television promotion on social media platforms are screenshots of highlights of the series and exquisite images of characters, the target video can be some highlight clips of the film and television series to be promoted on the Internet, the behind-the-scenes videos of the actors, and other videos. Of course, it can also be the original video clips of the original video of the film and television series, that is, the user can intercept the original video clips of the original video according to actual needs to generate a spliced ​​picture. It should be understood that the embodiments of the present disclosure do not limit the content and acquisition method of the target video. For example, the target video can be a video file transcoded from a video stream downloaded from the Internet, or it can be an original video file corresponding to the original video clip, etc.

[0045] In practical applications, video processing techniques known in the art can be used, for example, FFmpeg (an open source computer program for processing video and audio) software can be used to extract or intercept multiple video frames from the target video. Specifically, multiple video frames can be extracted from the target video in a manner of extracting (extracting or intercepting) one video frame every n seconds, or a specified number (such as 10 frames) of video frames can be uniformly extracted from the target video, and this is not limited by the embodiments of the present disclosure. Among them, the extracted multiple video frames can be saved in a temporary directory in a picture format such as .png format or .jpg format, so that the video frames can be used to generate a spliced ​​picture later.

[0046] In step S12, target boundary information of a main picture area in a target video is determined based on a plurality of video frames, where the target boundary information includes a target upper boundary position and a target lower boundary position of the main picture area.

[0047] Understandably, some target videos downloaded from the Internet may have a frame, such as Figure 2 As shown, the target video has a black upper frame and a lower frame. The upper frame may contain the video title, and the lower frame may contain information such as the title of the play. Figure 3 As shown, the target video may also only have a black upper frame, and the area where the main picture of the target video is located except the frame is the main picture area of ​​the target video. The picture content in the main picture area can be regarded as the effective picture content of the video. Therefore, by determining the target boundary information of the main picture area of ​​the target video, it is convenient to use the effective picture content in the main picture area to generate the required spliced ​​picture. Of course, there may be no frame in the target video, so the target boundary information of the main picture area in the target video can actually be the top boundary position and the bottom boundary position of the target video.

[0048] In practical applications, for example, the vertical pixel position (ie, the vertical pixel coordinate) can be used to represent the target upper boundary position and the target lower boundary position of the main image area. For example, Figure 4 As shown, Figure 2 A coordinate system is established with the upper left vertex of as the origin. The target upper boundary position of the main screen area can be represented by y1, and the target lower boundary position can be represented by y2.

[0049] In practical applications, for example, an artificial intelligence model can be trained to use the trained artificial intelligence model to determine the target boundary information of the main picture area of ​​the target video based on multiple video frames extracted from the target video.

[0050] Considering that the development cost of using an artificial intelligence model to determine target boundary information is high and the overall implementation efficiency may be low, the embodiment of the present disclosure provides a simpler and more efficient method for determining target boundary information. Specifically, the above method for determining target boundary information of a main screen area in a target video based on multiple video frames includes:

[0051] Step S121, for any video frame among the multiple video frames, by comparing the pixel value differences between adjacent pixels in each column in the video frame, the upper boundary position and the lower boundary position of each column are obtained, wherein the upper boundary position of each column includes the pixel position where the pixel value first suddenly changes when the pixel value differences of each column are compared in a top-to-bottom order, and the lower boundary position of each column includes the pixel position where the pixel value first suddenly changes when the pixel value differences of each column are compared in a bottom-to-top order;

[0052] Step S122, determining boundary information corresponding to the video frame according to the upper boundary position and the lower boundary position of each column in the video frame, the boundary information including the upper boundary position and the lower boundary position of the main picture area of ​​the video frame;

[0053] Step S123: determining target boundary information according to boundary information corresponding to each video frame in the plurality of video frames.

[0054] The pixel value may be, for example, an RGB color value, or the sum of RGB color values. Of course, a grayscale value, a brightness value, a contrast value, etc. may also be used, which is not limited in the embodiments of the present disclosure.

[0055] In step S121, by comparing the pixel value differences between adjacent pixels in each column of the video frame, the upper boundary position and the lower boundary position of each column are obtained, for example, it may include: comparing (i.e., traversing) the pixel value differences between adjacent pixels in each column in a top-to-bottom order, obtaining the pixel position where the pixel value first suddenly changes when comparing the pixel value differences in each column in a top-to-bottom order as the upper boundary position of each column, and comparing the pixel value differences between adjacent pixels in each column in a bottom-to-top order, obtaining the pixel position where the pixel value first suddenly changes when comparing the pixel value differences in each column in a bottom-to-top order as the lower boundary position of each column.

[0056] The first mutation of pixel values ​​can be understood as the first time that the pixel values ​​of two adjacent pixels are different, or the pixel values ​​differ greatly (such as greater than a certain threshold). It should be understood that the frame area in the video usually has a large area of ​​continuous identical pixel values. If the pixel values ​​of adjacent pixels in a column mutate for the first time, it is likely that they are traversed from the frame area to the main screen area. Therefore, by traversing the columns of the video frame and comparing the differences in the pixel values ​​of adjacent pixels, the position where the pixel value first mutates can be found, so as to determine the upper and lower boundary positions of each column.

[0057] Considering that the pixel values ​​of some pixels in the main image area that are adjacent to the frame area may be the same as the pixel values ​​of the pixels in the frame area, for example, some pixels in the main image area that are adjacent to the frame area are also black, or there may be some title texts of different colors in the frame area, such as the above Figure 2 and Figure 3 There is white title text in the black frame area, which will also cause a sudden change in pixel value. Therefore, in order to further accurately determine the upper boundary position and the lower boundary position of each column, the above method of finding the pixel position where the pixel first suddenly changes can be further improved. In a possible implementation method, the above step S121 compares the pixel value difference between adjacent pixels in each column of the video frame to obtain the upper boundary position and the lower boundary position of each column, which may include:

[0058] Step S1211, for the i-th column in the video frame, respectively, in a top-to-bottom order and a bottom-to-top order, compare the pixel value differences between adjacent pixels in the i-th column in the video frame to obtain the first pixel position where the pixel value first suddenly changes when the pixel value differences are compared in the top-to-bottom order and the second pixel position where the pixel value first suddenly changes when the pixel value differences are compared in the bottom-to-top order, where i is a positive integer;

[0059] Step S1212, starting from a row of pixels at a first pixel position, traverse upwards row by row to determine whether pixel values ​​of pixels in the same row are continuously the same, and starting from a row of pixels at a second pixel position, traverse downwards row by row to determine whether pixel values ​​of pixels in the same row are continuously the same, to obtain a third pixel position when pixel values ​​of pixels in the same row are continuously the same for the first time when traversing upwards from the first pixel position, and a fourth pixel position when pixel values ​​of pixels in the same row are continuously the same for the first time when traversing downwards from the second pixel position;

[0060] Step S1213: determine the third pixel position as the upper boundary position of the i-th column, and determine the fourth pixel position as the lower boundary position of the i-th column.

[0061] It should be understood that if the width and height of the video frame are known, then the number of vertical columns of pixels and horizontal rows of pixels in the video frame is known, or in other words, the number of pixels in each column and each row of the video frame is known.

[0062] For example, Figure 5 An exemplary video frame shown in FIG. 1 is used as an example to introduce the implementation of the above steps S1211 to S1213, wherein one grid represents one pixel point. Figure 5 There are 5 columns and 8 rows of pixels in total. For the first column b1 of the video frame, the pixel value difference between the second pixel in the first column and the first pixel can be compared in order from top to bottom, that is, to determine whether the pixel values ​​of the second pixel in the first column are the same as the first pixel or are less than the specified threshold. If they are the same or less than the specified threshold, it is considered that no pixel value mutation has occurred. Then, the pixel value difference between the third pixel in the first column and the second pixel is compared. If no pixel value mutation has occurred, the pixel value difference between the fourth pixel in the first column and the third pixel is then compared. If the pixel values ​​of the fourth pixel in the first column are different from the third pixel or are greater than the specified threshold, it is considered that the pixel value has mutated. At this time, the vertical pixel corresponding to the third pixel can be compared. The pixel position a3 in the vertical direction is determined as the first pixel position corresponding to the first column; at the same time, for the first column b1 of the video frame, the pixel value difference between the 7th pixel point and the 8th pixel point in the first column can be compared in order from bottom to top, that is, it is determined whether the pixel values ​​of the 7th pixel point and the 8th pixel point in the first column are the same or less than the specified threshold. If they are the same or less than the specified threshold, it is considered that no pixel value mutation occurs, and then the pixel value difference between the 6th pixel point and the 7th pixel point in the first column is compared. If the pixel values ​​of the 6th pixel point and the 7th pixel point in the first column are different or greater than the specified threshold, it is considered that the pixel value mutation occurs. At this time, the vertical pixel position a7 corresponding to the 7th pixel point can be determined as the second pixel position corresponding to the first column;

[0063] Then, for the first pixel position a3 corresponding to the first column, we can start from a row of pixels at the first pixel position a3 and traverse upwards row by row to see whether the pixel values ​​of the pixels in the same row are continuously the same. Specifically, we can first determine whether the pixel values ​​of a row of pixels corresponding to the first pixel position a3 (i.e., (b1, a3), (b2, a3), (b3, a3), (b4, a3), (b5, a3)) are continuously the same in order from left to right. If the pixel values ​​of (b1, a3) and (b2, a3) are the same, while the pixel values ​​of (b2, a3) and (b3, a3) are different, we can consider that the first pixel position a3 is not a pixel. Assume that the pixel values ​​of a row of pixels corresponding to a3 are not continuously the same, that is, there are pixels with sudden pixel values ​​in the a3th row. Then, we can judge whether the pixel values ​​of a row of pixels corresponding to a2 (that is, (b1, a2), (b2, a2), (b3, a2), (b4, a2), (b5, a2)) are continuously the same in order from left to right. If the pixel values ​​of (b1, a2), (b2, a2), (b3, a2), (b4, a2), (b5, a2)) are continuously the same, then a2 can be determined as the third pixel position corresponding to the first column (that is, the upper boundary position of the first column). At the same time, for the For the second pixel position a7 corresponding to column 1, we can first determine whether the pixel values ​​of a row of pixels corresponding to the second pixel position a7 (i.e., (b1, a7), (b2, a7), (b3, a7), (b4, a7), (b5, a7)) are continuously the same in order from left to right. If the pixel values ​​of (b1, a7), (b2, a7), (b3, a7), (b4, a7), (b5, a7) are continuously the same, then a7 can be directly determined as the third pixel position corresponding to column 1 (i.e., the lower boundary position of column 1). If the pixel values ​​of (b1, a7) and (b2, a7) are different, , it can be considered that the pixel values ​​of a row of pixels corresponding to the second pixel position a7 are not continuously the same, and then we can judge whether the pixel values ​​of a row of pixels corresponding to a8 (i.e., (b1, a8), (b2, a8), (b3, a8), (b4, a8), (b5, a8)) are continuously the same in order from left to right. If they are continuously the same, a8 can be determined as the third pixel position corresponding to the first column (i.e., the lower boundary position of the first column). Of course, since a8 is actually the bottom boundary position, it can be considered that the first column has no bottom frame, and the bottom boundary position of the first column is also the lower boundary position of the first column.

[0064] By analogy, for the second column b2 to the fifth column b5 of the video frame, the upper boundary position and the lower boundary position corresponding to each column in the second column b1 to the fifth column b5 can be obtained according to the determination method of the lower boundary position and the upper boundary position of the first column. In this way, it is possible to determine a more accurate upper boundary position and a lower boundary position of each column of the video frame.

[0065] In step S122, the boundary information corresponding to the video frame is determined according to the upper boundary position and the lower boundary position of each column in the video frame. For example, the average value of the multiple upper boundary positions corresponding to the multiple columns in the video frame can be used as the upper boundary position of the region of the main picture area of ​​the video frame, and the average value of the multiple lower boundary positions corresponding to the multiple columns in the video frame can be used as the lower boundary position of the region of the main picture area of ​​the video frame. In this way, the upper boundary position and the lower boundary position of the region can be randomly selected from all the columns of the video frame to calculate the upper boundary position and the lower boundary position of the region. Of course, the upper boundary position and the lower boundary position of the region can also be directly used to calculate the upper boundary position and the lower boundary position of the region, and the embodiments of the present disclosure are not limited to this.

[0066] Optionally, the pixel positions of the video frame from top to bottom can be set in ascending order, whereby, in the above step S122, the boundary information corresponding to the video frame is determined according to the upper boundary position and the lower boundary position of each column in the video frame, which can include: determining the maximum value among the multiple upper boundary positions corresponding to the multiple columns in the video frame as the regional upper boundary position of the main picture area of ​​the video frame; determining the minimum value among the multiple lower boundary positions corresponding to the multiple columns in the video frame as the regional lower boundary position of the main picture area of ​​the video frame. In this way, the upper boundary position and the lower boundary position of some columns can be randomly selected from all the columns of the video frame to determine the regional upper boundary position and the regional lower boundary position; of course, the regional upper boundary position and the regional lower boundary position can also be directly determined from the upper boundary position and the lower boundary position of all the columns of the video frame, and this is not limited in the embodiments of the present disclosure.

[0067] Alternatively, the pixel positions of the video frame from top to bottom may be set in descending order. Thus, when the pixel positions of the video frame from top to bottom are in descending order, in step S122, the boundary information corresponding to the video frame is determined according to the upper boundary position and the lower boundary position of each column in the video frame. This may include: determining the minimum value among the multiple upper boundary positions corresponding to the multiple columns in the video frame as the regional upper boundary position of the main picture area of ​​the video frame; and determining the maximum value among the multiple lower boundary positions corresponding to the multiple columns in the video frame as the regional lower boundary position of the main picture area of ​​the video frame.

[0068] Both methods aim to find the lowest upper boundary of each column and the highest lower boundary of each column. This can avoid misjudgment in each column and divide the frame content into the main screen area. Figure 4In the scenario shown, since there are texts in the upper and lower bounding boxes, in the column where the texts are located, pixel mutations may occur in the text area, causing the position of the texts to be used as the upper or lower boundary. By taking the lowermost of the upper boundaries of each column and the uppermost of the lower boundaries, such misjudgment can be effectively eliminated.

[0069] It should be understood that for each of the multiple video frames, the boundary information corresponding to each of the multiple video frames (i.e., the upper boundary position and the lower boundary position of the main screen area of ​​each video frame) can be obtained according to the implementation method of the above-mentioned steps S121 to S122, that is, multiple boundary information can be obtained, and then in step S123, the average value of the upper boundary positions of the regions in the multiple boundary information can be determined as the target upper boundary position of the main screen area in the target video, and the average value of the lower boundary positions of the regions in the multiple boundary information can be determined as the target lower boundary position of the main screen area in the target video. In this way, the boundary coordinates of the continuous color area can be automatically detected and extracted from the video frame, which is particularly suitable for processing video frames containing framed areas, so as to extract the effective main screen area from the video frame.

[0070] Considering that the main picture area (such as a picture of a dark sky) in some video frames may have a large area with pixel values ​​close to or the same as the frame area (such as a black frame), invalid boundary information may be obtained (that is, some video frames may not detect valid boundary information), and the invalid boundary information is usually outlier (that is, it is very different from the boundary information of other video frames). Therefore, in order to improve the accuracy of the target boundary information, in a possible implementation, in the above step S123, according to the boundary information corresponding to each video frame in the multiple video frames, determining the target boundary information may include:

[0071] Filtering outlier boundary information among the plurality of boundary information corresponding to the plurality of video frames to obtain a plurality of valid boundary information among the plurality of boundary information;

[0072] Determine the average value of the upper boundary positions of the regions in the multiple valid boundary information as the target upper boundary position of the main image region in the target video;

[0073] The average value of the lower boundary positions of the regions in the multiple valid boundary information is determined as the target lower boundary position of the main picture region in the target video.

[0074] In practical applications, those skilled in the art may use known statistical methods, such as the Fibonacci method, to integrate the boundary information of all video frames and filter out outlier boundary information among multiple boundary information, where the outlier boundary information is also invalid boundary information. Of course, other outlier data filtering methods known in the art may also be used, as long as the outlier boundary information can be screened and filtered out, and the embodiments of the present disclosure are not limited to this.

[0075] Among them, the boundary information remaining after filtering out the outlier boundary information from the multiple boundary information is also the effective boundary information. Then, the target upper boundary position of the main picture area in the target video can be obtained by calculating the average value of the upper boundary positions of the region in the multiple effective boundary information, and the target lower boundary position of the main picture area in the target video can be obtained by calculating the average value of the lower boundary positions of the region in the multiple effective boundary information, thereby realizing the use of the effective boundary information of multiple video frames to obtain unified and accurate target boundary information representing the entire target video, so as to facilitate subsequent cropping and other processing.

[0076] In step S13, each of the multiple video frames is cropped according to the target boundary information to obtain multiple subject area maps, where the subject area maps include the subject picture area in the video frame.

[0077] In practical applications, if the target boundary information is known, that is, the unified target upper boundary position and target lower boundary position of the main picture area are known, then the image cropping technology known in the art can be used to crop each of the multiple video frames according to the target boundary information to obtain multiple main area maps, which is equivalent to cropping the main picture area between the target upper boundary position and the target lower boundary position of each video frame to obtain the main area map. It should be understood that the embodiments of the present disclosure do not limit the cropping process of the video frame. For example, Figure 2 After cutting, we can get Figure 6a The main area diagram shown is Figure 3 After cutting, we can get Figure 6b A diagram of the subject area is shown.

[0078] In step S14, at least one dialogue set corresponding to at least one character in the target video is determined.

[0079] Among them, a single line set includes multiple lines associated with a single character in context and the display position of each line, and the multiple lines in a single line set are arranged according to the frame order of the video frame to which the main area map belongs.

[0080] In a possible implementation, the above-mentioned dialogue set may be obtained by performing text recognition on each of the plurality of main body area maps. Specifically, the above-mentioned determination of at least one dialogue set corresponding to at least one character in the target video may include:

[0081] Extracting the line data in each of the plurality of main area maps, the line data including the lines in the main area map and the display position of the lines;

[0082] Based on the speech data in each of the plurality of subject area maps, a global speech set is constructed, wherein the global speech set includes all the speech in the plurality of subject area maps arranged in a frame order of the video frames to which the subject area maps belong;

[0083] An artificial intelligence model is used to determine at least one line set corresponding to each role in at least one role based on the global line set.

[0084] Among them, text recognition technology known in the art, such as optical character recognition (OCR) technology, can be used to extract the lines in each main area map and the display position of the lines, and the embodiments of the present disclosure are not limited to this. In actual applications, before using OCR technology for text recognition, the main area map can be converted into data in a corresponding format (for example, it can be converted into Base64 encoding format) according to the format requirements of OCR technology for input data before OCR recognition is performed, and the embodiments of the present disclosure are not limited to this.

[0085] Among them, the display position of the lines can be understood as the position where the lines are displayed in the main area map. For example, the display position of the lines can be expressed as the relative height of the line area (that is, the display area of ​​the lines) relative to the bottom boundary of the main area map (also called the line height), or the display position of the lines can be expressed as the pixel distance from the highest point of the line area to the bottom boundary of the main area map; of course, it can also be expressed as the pixel coordinates of the line area in the main area map, etc., which is not limited to this embodiment of the present disclosure.

[0086] Considering that in addition to the lines, there may be some unfixed position flower characters in the main area map. For example, in some video screens, there may be a character profile (such as name, position, etc.) displayed when the character first appears, a scene introduction (such as place name) displayed when the scene first appears, and the title of the play, background introduction, inner monologue, etc. The display position of these flower characters is not fixed, but the lines and subtitles of the dialogue between the characters are usually displayed in the area specified at the bottom of the main screen area. Therefore, in order to ensure that the text extracted from the main area map is the lines, it is also possible to first intercept the line area map of the default line area at the bottom of the main area map based on the default line height (that is, the default height of the line display area relative to the bottom boundary), and perform text recognition on the intercepted line area map to obtain the line data in the main area map; of course, it can also be directly set to identify the default line area at the bottom of the main area map based on the default line height to obtain the line data in the main area map, and the embodiment of the present disclosure is not limited to this. It should be understood that the default line height can be a height preset based on experience, and the actual line height in the main area map may be less than or equal to the default line height.

[0087] In practical applications, after extracting the line data from each main area map, the lines extracted from multiple main area maps can be arranged in frame order according to the frame order of the video frames to which the main area maps belong in the target video to construct a global line set. Considering that the lines in multiple main area maps may also be repeated, it is also possible to first remove the lines with repeated content and the main area maps corresponding to the lines, and then arrange the remaining non-repeated lines in frame order to obtain a global line set. In this way, the artificial intelligence model can be provided with information on the order in which the lines appear, so as to facilitate the artificial intelligence model to accurately divide the order in which the lines appear into at least one line set corresponding to at least one role.

[0088] The artificial intelligence model can be implemented using the text-based "speaker segmentation" or "speaker recognition" technology in related technologies. These technologies can divide the text into different speakers, i.e., "roles", according to the context. The embodiments of the present disclosure do not limit the type, structure, training method, etc. of the artificial intelligence model, as long as it can achieve the required functions. For example, Figure 7 As shown, a global dialogue set can be extracted from the 8 main area maps. The lines in the global dialogue set can be filtered, grouped, sorted, etc. using an artificial intelligence model to obtain two dialogue sets for two characters (character A and character B).

[0089] It should be understood that the same character in the target video may have multiple lines of dialogue. For example, character A first says a line, then character B says a line, and then character A says another line. The two lines said by character A may not have a contextual association relationship. Therefore, the artificial intelligence model can divide each character into at least one group of multiple lines with contextual association based on the contextual relationship between the lines. That is, each character can correspond to at least one set of lines, such as the two lines of character A can be divided into two sets of lines.

[0090] In addition to identifying lines from the main area map, the required lines can also be extracted from an independent subtitle file. For example, as described above, the target video can be an original video clip of the original video of a film or TV series, and the original video is usually equipped with external subtitles. Therefore, at least one line set corresponding to at least one character in the target video can also be extracted from the subtitle file that matches the target video (i.e., the original video clip); it can be known that the subtitle file will contain information such as the content of the lines, the name of the character of the line content, the display position of the lines, and the display time. Therefore, based on the subtitle file, multiple lines with contextual associations for each character in the target video and the display position of each line can be extracted, and the frame order of the video frame to which each line belongs can also be known, thereby obtaining at least one line set for a single character. Of course, those skilled in the art can also use other known technical means in the art, such as speech recognition, etc., to determine at least one line set corresponding to at least one character in the target video, and the embodiments of the present disclosure are not limited to this.

[0091] In step S15, at least one spliced ​​image is generated based on at least one line set and the main area map corresponding to each line in each line set, and a single spliced ​​image includes the main screen area in the main area map corresponding to the first line in the single line set and the line area in the main area map corresponding to the remaining lines except the first line.

[0092] As described above, the multiple lines in the line set are arranged in the frame order of the video frames to which the main area map belongs. Therefore, the first line in the line set can be understood as the first line said by the character in the current dialogue. Then, the main area map to which the first line belongs can be spliced ​​with the line area maps in the main area maps to which the remaining lines except the first line belong to obtain a spliced ​​picture. Specifically, the generating of at least one spliced ​​picture according to at least one line set and the main area map corresponding to each line in each line set can include:

[0093] For any line set, according to the display position of at least one line other than the first line in the line set, the line area map of each remaining line in the main area map to which it belongs is cut out;

[0094] According to the frame order of the video frames to which the main area map corresponding to each line in the line set belongs, the main area map corresponding to the first line and the line area maps corresponding to the remaining lines are spliced ​​to obtain a spliced ​​image.

[0095] As described above, the display position of the lines can be expressed as the line height (i.e., the pixel distance from the highest point of the line area to the bottom border of the main area map). Therefore, the line area map of each line in the main area map to which the remaining lines belong can be cropped according to the display position of the remaining lines of each line, and the line area map includes the line area. Then, according to the frame order of the video frames to which the main area map corresponding to each line in the line set belongs (i.e., the arrangement order of each line in the line set), the main area map corresponding to the first line and the line area maps corresponding to the remaining lines can be spliced ​​in sequence to obtain a spliced ​​image. Exemplarily, for Figure 7 The two lines divided from the two sets can be obtained respectively Figure 8a and Figure 8b Two stitched pictures are shown.

[0096] The “one line” of dialogue here may refer to the dialogue that appears in the main area map of the same video frame.

[0097] It is taken into account that the dialogue set generated by the artificial intelligence model may also have a small number of lines or less content of lines (that is, the text length of the lines is short). For example, a certain dialogue set may have two lines: "Yes" and "Let's go". The lines in this kind of dialogue set can express less semantic information, which may not attract users to watch the drama, or it may not play a role in promoting the film and television drama. Therefore, it is also possible to select a dialogue set with a number of lines exceeding a certain threshold and / or a line content exceeding a certain threshold from at least one dialogue set generated in step S14 above, or it is also possible to select a dialogue set containing the largest number of lines and / or the longest line content to generate a spliced ​​image, and this is not limited to the embodiments of the present disclosure.

[0098] It should be understood that a line may appear in multiple consecutive video frames, that is, the first line may appear in at least two main area maps. At this time, a main area map can be randomly selected for stitching, or the facial area in each main area map to which the first line belongs can be identified and the clarity of the facial area can be evaluated to obtain the facial clarity score in each main area map, and the main area map with the highest facial clarity score can be selected to stitch the above-mentioned stitched image; the main area maps corresponding to the remaining lines are the intercepted line areas, so the main area maps corresponding to the remaining lines can be randomly selected, of course, they can also be selected according to the facial clarity score, and the embodiments of the present disclosure are not limited to this.

[0099] Taking into account that different frame extraction strategies (frame extraction methods, number of frames extracted, etc.) can be used when extracting multiple video frames from the target video, and a line may appear in multiple consecutive video frames, then multiple different spliced ​​images can also be generated for the same set of lines (for example, if the main area map of the first line is different, different spliced ​​images can be spliced). Therefore, the user can use different frame extraction strategies to execute the image generation method of the embodiment of the present disclosure multiple times for the same target video according to actual needs (such as the number of spliced ​​images to be generated), or can also execute the image generation method of the embodiment of the present disclosure separately for multiple different target videos to achieve batch generation of spliced ​​images, which is not limited by the embodiment of the present disclosure.

[0100] In practical applications, the generated stitched images can be uploaded to a storage server (such as a server using an object storage service (OSS)) for storage, so that the stitched images can be used to create promotional materials on various social platforms.

[0101] According to the image generation method of the embodiment of the present disclosure, by extracting multiple video frames of the target video, identifying the target upper boundary position and the target lower boundary of the main screen area, cropping out a main area map containing the main screen area in the video frame, and then determining at least one line set corresponding to at least one character in the target video, and generating at least one spliced ​​image according to the main screen area in the main area map corresponding to the first line of the lines in the line set and the line areas in the main area map corresponding to the remaining lines except the first line. It can be realized that context-related line splicing images divided by roles are automatically generated based on any target video, and low-cost and high-efficiency automatic batch production of high-quality spliced ​​images is realized. In an exemplary scenario, the spliced ​​images can be used as promotional materials for film and television dramas on social platforms.

[0102] As described above, the image sources used for promotion on social media platforms include exquisite images of people, so it is also possible to generate an image composed of high-quality and clear images of the characters in the target video. Therefore, in a possible implementation, the method may further include:

[0103] identifying a facial region in each of the plurality of subject region maps and evaluating the clarity of the facial region to obtain a facial clarity score for each subject region map;

[0104] Screening at least two target subject area maps from the multiple subject area maps according to the facial clarity score of each subject area map;

[0105] According to the aspect ratio of the main area image, at least two target main area images are horizontally or vertically spliced ​​to obtain a spliced ​​image, where the aspect ratio is the ratio between the width and the height of the main area image.

[0106] In practical applications, face recognition technology known in the art can be used to identify the facial area in each of the multiple subject area maps; and image clarity recognition technology known in the art can be used to evaluate the clarity of the facial area in each subject area map, which is not limited to the embodiments of the present disclosure.

[0107] The higher the facial clarity score of the character in the main body area map, the more exquisite the main body area map as a character image can be considered. Therefore, the facial clarity scores can be arranged from high to low, and the n main body area maps with the facial clarity scores ranked in the top n positions are selected as the target main body area maps to perform horizontal or vertical splicing on the n target main body area maps to obtain a spliced ​​image. Wherein, n can be, for example, 2 or 4, and a two-part puzzle or a four-part puzzle can be realized, and this is not limited in the embodiments of the present disclosure.

[0108] It should be understood that if the aspect ratio of the main area image is small, that is, the width of the main area image is smaller than the height, then the main area image is usually displayed as a vertical image. In this case, at least two target main area images can be horizontally spliced, for example, Figure 9a The stitched image is obtained by horizontal stitching as shown; if the width-to-height ratio of the main area image is large, that is, the width of the main area image is greater than the height, then the main area image is usually displayed as a horizontal image. At this time, at least two target main area images can be stitched vertically, for example, Figure 9b A stitched picture obtained by longitudinal stitching is shown.

[0109] In actual applications, after selecting at least two target subject area maps from multiple subject area maps, the at least two target subject area maps can also be fed back to the user through the front-end interface. A puzzle tool can also be provided in the front-end interface. The user can customize the splicing of the target subject area maps based on the puzzle tool to obtain a spliced ​​picture. The embodiments of the present disclosure do not limit this.

[0110] In actual applications, the above-mentioned generated target main area map or the spliced ​​image formed by splicing the target main area map can be uploaded to a storage server (such as a server using an object storage service (OSS)) for storage, so as to facilitate the subsequent use of the spliced ​​image or directly use the target main area map to create promotional materials on various social platforms.

[0111] According to the embodiments of the present disclosure, a subject area map can be used to automatically generate a spliced ​​image of a character with high facial definition at low cost and high efficiency.

[0112] As described above, some target videos have a top frame, and there are usually some title texts in the top frame area. These title texts can also be used as promotional materials on social platforms. Therefore, in a possible implementation, when the target upper boundary position is different from the top boundary of the video frame, the method may further include:

[0113] For any video frame, the title area in the video frame is cropped according to the target upper boundary position to obtain a title area map of the video frame;

[0114] Perform text recognition on the title area image to obtain the video title of the video frame; the video title is used to assemble promotional materials on social platforms with spliced ​​pictures.

[0115] It should be understood that if the target upper boundary position of the main picture area is different from the top boundary of the video frame (which can be understood as the maximum upper boundary of the video frame), it can be considered that there is a top frame area (that is, the top title area) in the video frame. At this time, the title area from the target upper boundary position to the top boundary of the video frame can be cropped according to the target upper boundary position to obtain a title area map; then, text recognition technology known in the art, such as OCR technology, can be used to perform text recognition on the title area map to obtain the video title of the video frame, which is not limited to this embodiment of the present disclosure.

[0116] In practical applications, the extracted video titles can be stored in a database as promotional materials, so that users can use the video titles and any of the above-mentioned spliced ​​pictures to compose promotional materials on social platforms.

[0117] Based on the above picture generation method provided by the embodiment of the present disclosure, the embodiment of the present disclosure also provides Fig.10 A schematic diagram of a picture generation process is shown, such as Fig.10As shown, the image generation process includes: obtaining a target video (i.e., a video file); extracting 10 video frames from the target video; determining the width and height information of the video frame (including the height and width of the video frame) and the target boundary information of the main screen area (the target upper boundary position y1 and the target lower boundary position y2); based on the height, y1, and y2 of the video frame, a main area map of the video frame can be intercepted, and facial recognition and clarity recognition are performed (i.e., the facial area in each of the multiple main area maps is identified and the clarity of the facial area is evaluated), and at least two target main area maps are screened out based on the facial clarity score, and the at least two target main area maps are further binary spliced ​​or quadrupled, etc., to obtain a binary spliced ​​image or a quadrupled image. The images are stitched together and stored in the OSS server; and, based on y1 and y2, a dialogue area map at the bottom of the main screen area of ​​the video frame can be captured, and the dialogue area map can be subjected to OCR recognition to obtain the dialogue data in each main screen area; then, the dialogue context can be filtered using an artificial intelligence model and divided into at least one dialogue set according to the role, and the complete main area map of the first line of the dialogue set and the dialogue area maps of other lines can be selected to obtain a context stitching image (that is, a dialogue stitching image) and store it in the OSS server; the title area of ​​the video frame can also be captured based on y1, and the title area map can be subjected to OCR recognition to obtain the video title; the video title and dialogue data can be stored in a database for use as promotional materials.

[0118] According to the image generation method of the embodiment of the present invention, high-quality spliced ​​images can be produced for use in drama promotion and marketing, and can be automatically mass-produced to reduce labor costs; it can support secondary creation of drama videos (that is, generating dialogue puzzles, character puzzles, etc.); it can use the promotion and marketing needs of film and television content as the background to lower the threshold for creating promotion and marketing materials on social platforms and reduce production costs, and it realizes a high-quality image production process assisted by artificial intelligence models and image processing technology, which greatly shortens the time to produce images and is conducive to increasing the attractiveness of drama promotion and marketing.

[0119] According to the picture generation method of the embodiment of the present disclosure, it is possible to use video frame extraction and intelligent image analysis technology to efficiently and accurately extract the main picture and line data, combine the artificial intelligence model to intelligently group the lines, and innovatively realize the mass production of line splicing pictures and character splicing pictures, greatly improving the efficiency of picture production; specifically, by using video frame extraction technology and using RGB values ​​for boundary judgment, it is possible to more accurately judge whether the pixel points are continuous, thereby improving the efficiency of finding continuous pixel points and stably obtaining the effective main picture area of ​​the video frame; and, by extracting the line area and title area of ​​the picture through the target boundary information of the main picture area, using OCR technology to obtain the lines and titles, and performing face recognition and clarity recognition on the main area map, the picture quality of the spliced ​​picture can be improved. By using the artificial intelligence model to group the lines according to the context and role, the line picture splicing effect can be achieved, which conforms to the splicing picture style of the promotional materials of the real scene.

[0120] Fig.11 A block diagram of a picture generating device according to an embodiment of the present disclosure is shown as follows: Fig.11 As shown, the device comprises:

[0121] An extraction module 111 is used to obtain a target video and extract multiple video frames from the target video;

[0122] A boundary determination module 112, configured to determine target boundary information of a main picture area in the target video based on the multiple video frames, wherein the target boundary information includes a target upper boundary position and a target lower boundary position of the main picture area;

[0123] A cropping module 113, configured to crop each of the plurality of video frames according to the target boundary information to obtain a plurality of subject area maps, wherein the subject area maps include a subject picture area in the video frame;

[0124] A dialogue determination module 114 is used to determine at least one dialogue set corresponding to at least one character in the target video; wherein a single dialogue set includes multiple dialogues associated with a single character in context and a display position of each dialogue, and the multiple dialogues in the single dialogue set are arranged according to the frame order of the video frame to which the main area map belongs;

[0125] The image generation module 115 is used to generate at least one spliced ​​image based on the at least one line set and the main area map corresponding to each line in each line set, wherein a single spliced ​​image includes the main screen area in the main area map corresponding to the first line in the single line set and the line area in the main area map corresponding to the remaining lines except the first line.

[0126] In a possible implementation, the target boundary information of the main picture area in the target video is determined based on the multiple video frames, including: for any video frame in the multiple video frames, by comparing the pixel value differences between adjacent pixels in each column in the video frame, the upper boundary position of each column includes the pixel position where the pixel value first mutates when the pixel value differences are compared in a top-to-bottom order in each column, and the lower boundary position of each column includes the pixel position where the pixel value first mutates when the pixel value differences are compared in a bottom-to-top order in each column; according to the upper boundary position and the lower boundary position of each column in the video frame, the boundary information corresponding to the video frame is determined, the boundary information including the regional upper boundary position and the regional lower boundary position of the main picture area of ​​the video frame; according to the boundary information corresponding to each video frame in the multiple video frames, the target boundary information is determined.

[0127] In a possible implementation, the obtaining of the upper boundary position and the lower boundary position of each column by comparing the pixel value differences of adjacent pixels in each column of the video frame includes: for the i-th column in the video frame, comparing the pixel value differences between adjacent pixels in the i-th column in the video frame in a top-to-bottom order and a bottom-to-top order, respectively, to obtain the first pixel position where the pixel value of the i-th column first undergoes a sudden change when the pixel value differences are compared in a top-to-bottom order and the second pixel position where the pixel value first undergoes a sudden change when the pixel value differences are compared in a bottom-to-top order; starting from a row of pixels at the first pixel position, Whether the pixel values ​​of the pixels in the same row are continuously the same when traversing upward row by row from the first pixel position, and whether the pixel values ​​of the pixels in the same row are continuously the same when traversing downward row by row from the second pixel position, obtain the third pixel position when the pixel values ​​of the pixels in the same row are continuously the same for the first time when traversing upward from the first pixel position, and the fourth pixel position when the pixel values ​​of the pixels in the same row are continuously the same for the first time when traversing downward from the second pixel position; determine the third pixel position as the upper boundary position of the i-th column, and determine the fourth pixel position as the lower boundary position of the i-th column.

[0128] In a possible implementation, the pixel positions of the video frame from top to bottom are in ascending order, and the boundary information corresponding to the video frame is determined based on the upper boundary position and the lower boundary position of each column in the video frame, including: determining the maximum value of multiple upper boundary positions corresponding to multiple columns in the video frame as the regional upper boundary position of the main picture area of ​​the video frame; determining the minimum value of multiple lower boundary positions corresponding to multiple columns in the video frame as the regional lower boundary position of the main picture area of ​​the video frame.

[0129] In a possible implementation, the target boundary information is determined based on the boundary information corresponding to each video frame in the multiple video frames, including: filtering outlier boundary information in the multiple boundary information corresponding to the multiple video frames to obtain multiple valid boundary information in the multiple boundary information; determining the average value of the upper boundary positions of the region in the multiple valid boundary information as the target upper boundary position of the main picture region in the target video; and determining the average value of the lower boundary positions of the region in the multiple valid boundary information as the target lower boundary position of the main picture region in the target video.

[0130] In one possible implementation, determining at least one line set corresponding to at least one character in the target video includes: extracting line data in each of the multiple main area maps, the line data including the lines in the main area map and the display position of the lines; constructing a global line set based on the line data in each of the multiple main area maps, the global line set including all lines in the multiple main area maps arranged in the frame order of the video frames to which the main area maps belong; and determining at least one line set corresponding to each of at least one character according to the global line set using an artificial intelligence model.

[0131] In a possible implementation, generating at least one spliced ​​image based on the at least one line set and the main area map corresponding to each line in each line set includes: for any line set, according to the display position of at least one remaining line in the line set except the first line, cutting out the line area map of each remaining line in the main area map to which they belong; and splicing the main area map corresponding to the first line with the line area maps corresponding to the remaining lines in accordance with the frame order of the video frames to which the main area map corresponding to each line in the line set belongs, to obtain a spliced ​​image.

[0132] In a possible implementation, the device also includes: a scoring module, used to identify the facial area in each of the multiple body area images and evaluate the clarity of the facial area to obtain a facial clarity score for each body area image; a screening module, used to screen out at least two target body area images from the multiple body area images according to the facial clarity score of each body area image; and a splicing module, used to perform horizontal or vertical splicing of the at least two target body area images according to the aspect ratio of the body area images to obtain a spliced ​​image, wherein the aspect ratio is the ratio between the width and the height of the body area image.

[0133] In a possible implementation, when the target upper boundary position is different from the top boundary position of the video frame, the device further includes: a title cropping module, which is used to crop the title area in the video frame for any video frame according to the target upper boundary position to obtain a title area map of the video frame; a title recognition module, which is used to perform text recognition on the title area map to obtain a video title of the video frame; the video title is used to assemble promotional materials on the social platform together with the spliced ​​picture.

[0134] According to the embodiments of the present disclosure, multiple video frames of a target video are extracted, the target upper boundary position and the target lower boundary of the main screen area are identified, a main area map containing the main screen area in the video frame is cropped, and then at least one line set corresponding to at least one character in the target video is determined, and at least one spliced ​​image is generated according to the main screen area in the main area map corresponding to the first line of the lines in the line set and the line areas in the main area map corresponding to the remaining lines except the first line. This can realize automatic generation of context-related line spliced ​​images divided by roles based on any target video, and realize low-cost and high-efficiency automatic batch production of high-quality spliced ​​images. In an exemplary scenario, the spliced ​​images can be used as promotional materials for film and television dramas on social platforms.

[0135] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0136] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0137] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0138] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0139] Fig.12 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig.12 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0140] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0141] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0142] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0143] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0144] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0145] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0146] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0147] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0148] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0149] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0150] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating an image, characterized in that: include: Acquire a target video, and extract multiple video frames from the target video; Based on the multiple video frames, determine target boundary information of a main picture area in the target video, wherein the target boundary information includes a target upper boundary position and a target lower boundary position of the main picture area; According to the target boundary information, each of the multiple video frames is cropped to obtain multiple subject area maps, wherein the subject area map includes a subject picture area in the video frame; Determine at least one dialogue set corresponding to at least one character in the target video; wherein a single dialogue set includes multiple dialogues associated with a single character in context and a display position of each dialogue, and the multiple dialogues in the single dialogue set are arranged according to the frame order of the video frame to which the main area map belongs; At least one spliced ​​image is generated based on the at least one line set and the main area map corresponding to each line in each line set, wherein a single spliced ​​image includes the main screen area in the main area map corresponding to the first line in the single line set and the line areas in the main area map corresponding to the remaining lines except the first line.

2. The method according to claim 1, characterized in that The determining target boundary information of the main picture area in the target video based on the multiple video frames includes: For any video frame among the multiple video frames, by comparing the pixel value differences between adjacent pixels in each column of the video frame, an upper boundary position and a lower boundary position of each column are obtained, wherein the upper boundary position of each column includes the pixel position where the pixel value first suddenly changes when the pixel value differences of each column are compared in a top-to-bottom order, and the lower boundary position of each column includes the pixel position where the pixel value first suddenly changes when the pixel value differences of each column are compared in a bottom-to-top order; Determine boundary information corresponding to the video frame according to the upper boundary position and the lower boundary position of each column in the video frame, wherein the boundary information includes the upper boundary position and the lower boundary position of the main screen area of ​​the video frame; The target boundary information is determined according to the boundary information corresponding to each video frame in the multiple video frames.

3. The method according to claim 2, characterized in that The step of obtaining the upper boundary position and the lower boundary position of each column by comparing the pixel value differences of adjacent pixels in each column of the video frame includes: For the i-th column in the video frame, respectively, in a top-to-bottom order and a bottom-to-top order, compare the pixel value differences between adjacent pixels in the i-th column in the video frame to obtain a first pixel position where a pixel value mutation first occurs when comparing the pixel value differences in the top-to-bottom order and a second pixel position where a pixel value mutation first occurs when comparing the pixel value differences in the bottom-to-top order; Whether pixel values ​​of pixels in the same row are continuously the same when traversing upward from a row of pixels at the first pixel position, and whether pixel values ​​of pixels in the same row are continuously the same when traversing downward from a row of pixels at the second pixel position, obtain a third pixel position when pixel values ​​of pixels in the same row are continuously the same for the first time when traversing upward from the first pixel position, and a fourth pixel position when pixel values ​​of pixels in the same row are continuously the same for the first time when traversing downward from the second pixel position; The third pixel position is determined as the upper boundary position of the i-th column, and the fourth pixel position is determined as the lower boundary position of the i-th column.

4. The method according to claim 2 or 3, characterized in that: The pixel positions of the video frame are in ascending order from the top to the bottom, and the determining of the boundary information corresponding to the video frame according to the upper boundary position and the lower boundary position of each column in the video frame includes: Determine the maximum value among a plurality of upper boundary positions corresponding to a plurality of columns in the video frame as the region upper boundary position of the main screen region of the video frame; The minimum value among a plurality of lower boundary positions corresponding to a plurality of columns in the video frame is determined as the region lower boundary position of the main picture region of the video frame.

5. The method according to claim 2 or 3, characterized in that: The determining the target boundary information according to the boundary information corresponding to each video frame in the plurality of video frames includes: Filtering outlier boundary information among the plurality of boundary information corresponding to the plurality of video frames to obtain a plurality of valid boundary information among the plurality of boundary information; Determine the average value of the upper boundary positions of the regions in the multiple valid boundary information as the target upper boundary position of the main picture region in the target video; An average value of the lower boundary positions of the regions in the plurality of valid boundary information is determined as a target lower boundary position of the main picture region in the target video.

6. The method according to claim 1, characterized in that The determining of at least one dialogue set corresponding to at least one character in the target video includes: Extracting the line data in each of the plurality of main body area maps, the line data including the lines in the main body area map and the display position of the lines; Based on the speech data in each of the plurality of subject area maps, a global speech set is constructed, wherein the global speech set includes all the speech in the plurality of subject area maps arranged in a frame order of the video frames to which the subject area maps belong; An artificial intelligence model is used to determine at least one line set corresponding to each of at least one character based on the global line set.

7. The method according to claim 1, characterized in that The step of generating at least one spliced ​​image according to the at least one line set and the main area map corresponding to each line in each line set includes: For any line set, according to the display position of at least one line other than the first line in the line set, the line area map of each remaining line in the main area map to which it belongs is cut out; According to the frame order of the video frames to which the main area map corresponding to each line in the line set belongs, the main area map corresponding to the first line and the line area maps corresponding to the remaining lines are spliced ​​to obtain a spliced ​​image.

8. The method according to claim 1, characterized in that The method further comprises: Identifying a facial region in each of the plurality of subject region maps and evaluating the clarity of the facial region to obtain a facial clarity score for each subject region map; Screening out at least two target subject area images from the plurality of subject area images according to the facial clarity score of each subject area image; According to the aspect ratio of the main area image, the at least two target main area images are horizontally spliced ​​or vertically spliced ​​to obtain a spliced ​​image, wherein the aspect ratio is the ratio between the width and the height of the main area image.

9. The method according to claim 1, characterized in that: In the case where the target upper boundary position is different from the top boundary position of the video frame, the method further comprises: For any video frame, according to the target upper boundary position, the title area in the video frame is cropped to obtain a title area map of the video frame; Text recognition is performed on the title area image to obtain a video title of the video frame; the video title is used to compose promotional materials on a social platform with the spliced ​​image.

10. A picture generating device, characterized in that: include: An extraction module, used to obtain a target video and extract multiple video frames from the target video; A boundary determination module, configured to determine target boundary information of a main picture area in the target video based on the multiple video frames, wherein the target boundary information includes a target upper boundary position and a target lower boundary position of the main picture area; A cropping module, configured to crop each of the plurality of video frames according to the target boundary information to obtain a plurality of subject area maps, wherein the subject area map includes a subject picture area in the video frame; A dialogue determination module is used to determine at least one dialogue set corresponding to at least one character in the target video; wherein a single dialogue set includes multiple dialogues associated with a single character in context and a display position of each dialogue, and the multiple dialogues in the single dialogue set are arranged according to the frame order of the video frame to which the main area map belongs; The image generation module is used to generate at least one spliced ​​image based on the at least one line set and the main area map corresponding to each line in each line set, wherein a single spliced ​​image includes a main screen area in the main area map corresponding to the first line in the single line set and a line area in the main area map corresponding to the remaining lines except the first line.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 9 when executing the instructions stored in the memory.

12. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images