Image generation method and device, electronic device, and storage medium

Abstract

The present disclosure relates to an image generation method and apparatus, an electronic device, and a storage medium. The method comprises: extracting multiple video frames from a target video; determining target boundary information of a main screen area in the target video, the target boundary information including a target upper boundary position and a target lower boundary position of the main screen area; cropping each of the multiple video frames based on the target boundary information to obtain multiple main screen area maps; determining at least one dialogue set corresponding to at least one character in the target video; and generating at least one spliced image based on the at least one dialogue set and the main screen area map corresponding to each line in each dialogue set, wherein a single spliced image includes the main screen area in the main screen area map corresponding to the first line in the single dialogue set and the dialogue areas in the main screen area map corresponding to the remaining lines except the first line. Thus, high-quality spliced images can be automatically and efficiently produced in batches at low cost.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a method and device for generating an image, an electronic device, and a storage medium. Background Art

[0002] In the film and television industry, film and television publicity and promotion are a very critical link, and can even determine the market performance of a work to a certain extent. Its importance is reflected in many aspects: carefully planned publicity and promotion strategies in the early stage can create popularity for the work in advance and stimulate the audience's curiosity and anticipation; attracting the audience's attention through various forms of promotion such as trailers, posters, and stills can effectively increase public awareness of film and television works; through precise positioning of content and market segmentation during the publicity and promotion process, and using popular short video platforms and social media platforms for promotion, it can help the work reach the target audience more effectively.

[0003] Currently, promotional materials for film and television content on social media platforms are usually produced manually (such as dialogue splicing pictures, character splicing pictures, etc.). In particular, pictures are usually positively correlated with the number of readers and interactions of the published content, but the manual production method is costly and inefficient. Summary of the Invention

[0004] In view of this, the present disclosure proposes a method and device for generating an image, an electronic device, and a storage medium, which can automatically batch-produce high-quality spliced images at low cost and high efficiency.

[0005] According to one aspect of the present disclosure, a method for generating an image is provided, comprising: acquiring a target video and extracting multiple video frames from the target video; determining target boundary information of a main screen area in the target video based on the multiple video frames, the target boundary information including a target upper boundary position and a target lower boundary position of the main screen area; cropping each of the multiple video frames based on the target boundary information to obtain multiple main screen area maps, the main screen areas in the video frames; determining at least one line set corresponding to at least one character in the target video; wherein a single line set includes multiple lines associated with a single character in a context and a display position of each line, and the multiple lines in the single line set are arranged in a frame order of the video frames to which the main screen area maps belong; and generating at least one spliced image based on the at least one line set and the main screen area maps corresponding to each line in each line set, the single spliced image including the main screen area in the main screen area map corresponding to a first line in the single line set and the line areas in the main screen area map corresponding to the remaining lines except the first line.

[0006] In a possible implementation, the target boundary information of the main picture area in the target video is determined based on the multiple video frames, including: for any video frame among the multiple video frames, by comparing the pixel value differences between adjacent pixels in each column of the video frame, the upper boundary position and the lower boundary position of each column are obtained, the upper boundary position of each column includes the pixel position where the pixel value first mutates when the pixel value differences of each column are compared in a top-down order, and the lower boundary position of each column includes the pixel position where the pixel value first mutates when the pixel value differences of each column are compared in a bottom-up order; according to the upper boundary position and the lower boundary position of each column in the video frame, the boundary information corresponding to the video frame is determined, the boundary information including the upper boundary position and the lower boundary position of the area of the main picture area of the video frame; according to the boundary information corresponding to each video frame among the multiple video frames, the target boundary information is determined.

[0007] In a possible implementation, obtaining the upper boundary position and the lower boundary position of each column by comparing the pixel value differences of adjacent pixels in each column of the video frame includes: for the i-th column in the video frame, comparing the pixel value differences between adjacent pixels in the i-th column in the video frame in a top-to-bottom order and a bottom-to-top order, respectively, to obtain the first pixel position of the i-th column where the pixel value first suddenly changes when the pixel value differences are compared in a top-to-bottom order and the second pixel position where the pixel value first suddenly changes when the pixel value differences are compared in a bottom-to-top order; starting from a row of pixels at the first pixel position, Whether the pixel values of the pixels in the same row are continuously the same when traversing upward row by row from the first pixel position, and whether the pixel values of the pixels in the same row are continuously the same when traversing downward row by row from the second pixel position are continuously the same, obtain the third pixel position when the pixel values of the pixels in the same row are continuously the same for the first time when traversing upward from the first pixel position, and the fourth pixel position when the pixel values of the pixels in the same row are continuously the same for the first time when traversing downward from the second pixel position; determine the third pixel position as the upper boundary position of the i-th column, and determine the fourth pixel position as the lower boundary position of the i-th column.

[0008] In one possible implementation, the pixel positions of the video frame are in ascending order from top to bottom, and the boundary information corresponding to the video frame is determined based on the upper boundary position and the lower boundary position of each column in the video frame, including: determining the maximum value of multiple upper boundary positions corresponding to multiple columns in the video frame as the regional upper boundary position of the main picture area of the video frame; and determining the minimum value of multiple lower boundary positions corresponding to multiple columns in the video frame as the regional lower boundary position of the main picture area of the video frame.

[0009] In a possible implementation, the target boundary information is determined based on the boundary information corresponding to each video frame in the multiple video frames, including: filtering the outlier boundary information in the multiple boundary information corresponding to the multiple video frames to obtain multiple valid boundary information in the multiple boundary information; determining the average value of the upper boundary positions of the region in the multiple valid boundary information as the target upper boundary position of the main picture area in the target video; and determining the average value of the lower boundary positions of the region in the multiple valid boundary information as the target lower boundary position of the main picture area in the target video.

[0010] In one possible implementation, determining at least one line set corresponding to at least one character in the target video includes: extracting line data from each of the multiple main area maps, the line data including the lines in the main area map and the display position of the lines; constructing a global line set based on the line data in each of the multiple main area maps, the global line set including all lines in the multiple main area maps arranged in the frame order of the video frames to which the main area maps belong; and determining at least one line set corresponding to each of at least one character based on the global line set using an artificial intelligence model.

[0011] In one possible implementation, at least one spliced image is generated based on the at least one line set and the main area map corresponding to each line in each line set, including: for any line set, based on the display position of at least one remaining line in the line set except the first line, cropping the line area map of each remaining line in the main area map to which it belongs; and splicing the main area map corresponding to the first line with the line area maps corresponding to the remaining lines in accordance with the frame order of the video frames to which the main area map corresponding to each line in the line set belongs, to obtain a spliced image.

[0012] In one possible implementation, the method further includes: identifying a facial area in each of the multiple subject area maps and evaluating the clarity of the facial area to obtain a facial clarity score for each subject area map; screening at least two target subject area maps from the multiple subject area maps based on the facial clarity score of each subject area map; and horizontally or vertically splicing the at least two target subject areas based on the aspect ratio of the subject area maps to obtain a spliced image, where the aspect ratio is the ratio between the width and height of the subject area map.

[0013] In a possible implementation, when the target upper boundary position is different from the top boundary position of the video frame, the method further includes: for any video frame, according to the target upper boundary position, cropping the title area in the video frame to obtain a title area map of the video frame; performing text recognition on the title area map to obtain a video title of the video frame; the video title is used to form promotional materials on a social platform together with the spliced image.

[0014] According to another aspect of the present disclosure, a picture generation device is provided, comprising: an extraction module for acquiring a target video and extracting a plurality of video frames from the target video; a boundary determination module for determining target boundary information of a main screen area in the target video based on the plurality of video frames, the target boundary information including a target upper boundary position and a target lower boundary position of the main screen area; a cropping module for cropping each of the plurality of video frames according to the target boundary information to obtain a plurality of main screen area maps, the main screen area maps including the main screen area in the video frame; a line determination module for determining the target At least one line set corresponding to at least one character in a video; wherein a single line set includes multiple lines associated with a single character in context and the display position of each line, and the multiple lines in the single line set are arranged according to the frame order of the video frames to which the main area map belongs; an image generation module, for generating at least one spliced image based on the at least one line set and the main area map corresponding to each line in each line set, the single spliced image including the main screen area in the main area map corresponding to the first line in the single line set and the line areas in the main area map corresponding to the remaining lines except the first line.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0016] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0018] According to various aspects of the present disclosure, by extracting multiple video frames of a target video, identifying the target upper boundary position and the target lower boundary of the main screen area, cropping out a main area map containing the main screen area in the video frame, and then determining at least one line set corresponding to at least one character in the target video, and generating at least one spliced image according to the main screen area in the main area map corresponding to the first line of the lines in the line set and the line areas in the main area map corresponding to the remaining lines except the first line, it is possible to automatically generate context-related line splicing images divided by roles based on any target video, and realize low-cost and high-efficiency automatic batch production of high-quality spliced images. In an exemplary scenario, the spliced images can be used as promotional materials for film and television dramas on social platforms.

[0019] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0021] Figure 1 A flowchart of a method for generating an image according to an embodiment of the present disclosure is shown.

[0022] Figure 2 A schematic diagram of a frame of a target video according to an embodiment of the present disclosure is shown.

[0023] Figure 3 A schematic diagram of another frame of a target video according to an embodiment of the present disclosure is shown.

[0024] Figure 4 A schematic diagram illustrating target boundary information according to an embodiment of the present disclosure.

[0025] Figure 5 A schematic diagram showing an exemplary video frame according to an embodiment of the present disclosure.

[0026] Figure 6a A schematic diagram showing a main area map according to an embodiment of the present disclosure is shown.

[0027] Figure 6b A schematic diagram illustrating another main area diagram according to an embodiment of the present disclosure is shown.

[0028] Figure 7 A schematic diagram illustrating a process for determining a line set according to an embodiment of the present disclosure is shown.

[0029] Figure 8a and Figure 8bA schematic diagram showing two stitched pictures according to an embodiment of the present disclosure is shown.

[0030] Figure 9a A schematic diagram showing a stitched image obtained by horizontal stitching according to an embodiment of the present disclosure is shown.

[0031] Figure 9b A schematic diagram showing a stitched image obtained by vertical stitching according to an embodiment of the present disclosure is shown.

[0032] Figure 10 A schematic diagram illustrating a picture generation process according to an embodiment of the present disclosure is shown.

[0033] Figure 11 A block diagram of a picture generating device according to an embodiment of the present disclosure is shown.

[0034] Figure 12 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0036] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0037] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C. In the description of this disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0038] It should be understood that the terms "first," "second," and the like in the claims, specification, and drawings of the present disclosure are used to distinguish between different objects, rather than to describe a specific order. The terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0039] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0040] The image generation method of the embodiment of the present disclosure can be deployed on various terminal devices through software or hardware modification. The terminal device involved in the embodiment of the present disclosure may refer to a device with a wireless connection function and / or a wired connection function. The wireless connection function refers to the ability to connect to other devices through wireless connection methods such as wifi and Bluetooth. The terminal device involved in the embodiment of the present disclosure may also communicate with other devices through a wired connection function. The terminal device involved in the embodiment of the present disclosure may be a touch screen, a non-touch screen, or a screenless device. The touch screen device can be controlled by clicking, sliding, etc. on the display screen with a finger or a stylus. The non-touch screen device can be connected to an input device such as a mouse, keyboard, touch panel, and the terminal device can be controlled through the input device. For example, a device without a screen can be a Bluetooth speaker without a screen. For example, the terminal device of the present application may include but is not limited to user equipment (UE), mobile device, user terminal, terminal, handheld device, tablet computer, laptop computer, PDA, computing device, etc.

[0041] The image generation method of the embodiment of the present disclosure can also be deployed on a server. The server can be located in the cloud or locally. It can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. It can refer to a device with a wireless connection function. The wireless connection function means that it can be connected to other servers or terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The server involved in the embodiment of the present disclosure may also have the function of communicating through a wired connection. For example, the server of the embodiment of the present disclosure can be located in the cloud, communicate with the terminal device, receive the target video sent by the terminal device, and use the image generation method deployed on the server to generate at least one spliced picture based on the target video, and return it to the terminal device to display the generated at least one spliced picture to the user in the terminal device.

[0042] Figure 1 FIG. 1 is a flow chart showing a method for generating an image according to an embodiment of the present disclosure. The method can be applied to the above-mentioned terminal device or server. Figure 1 As shown, the method includes: steps S11 to S15.

[0043] In step S11 , a target video is acquired, and multiple video frames in the target video are extracted.

[0044] Considering that most of the images used for film and television promotion on social media platforms are screenshots of highlights from the series and exquisite images of characters, the target video can be some online videos of highlights of the film and television series to be promoted, videos of actors shooting behind-the-scenes, etc. Of course, it can also be the original video clips of the original video of the film and television series. That is, users can intercept the original video clips of the original video according to actual needs to generate a spliced image. It should be understood that the embodiments of the present disclosure do not limit the content and acquisition method of the target video. For example, the target video can be a video file transcoded from a video stream downloaded from the Internet, or it can be the original video file corresponding to the original video clip, etc.

[0045] In practical applications, video processing techniques known in the art can be used, for example, FFmpeg (an open source computer program for processing video and audio) software can be used to extract or intercept multiple video frames from the target video. Specifically, multiple video frames can be extracted from the target video in a manner of extracting (extracting or intercepting) one video frame every n seconds, or a specified number (such as 10 frames) of video frames can be uniformly extracted from the target video. This is not limited in the embodiments of the present disclosure. The extracted multiple video frames can be saved as image formats such as .png format or .jpg format in a temporary directory to facilitate the subsequent use of these video frames to generate a spliced image.

[0046] In step S12 , target boundary information of a main picture area in a target video is determined based on multiple video frames, where the target boundary information includes a target upper boundary position and a target lower boundary position of the main picture area.

[0047] Understandably, some target videos downloaded from the Internet may have a frame, such as Figure 2 As shown, the target video has a black upper frame and a lower frame. The upper frame can contain the video title, and the lower frame can contain information such as the title of the drama. Figure 3 As shown, the target video may also only have a black upper frame. The area of the target video where the main image is located outside the frame is the main image area of the target video. The image content within the main image area can be regarded as the effective image content of the video. Therefore, by determining the target boundary information of the main image area of the target video, it is convenient to subsequently use the effective image content within the main image area to generate the required spliced image. Of course, the target video may also have no frame. In this case, the target boundary information of the main image area of the target video can actually be the top and bottom boundary positions of the target video.

[0048] In practical applications, for example, the vertical pixel position (ie, the vertical pixel coordinate) can be used to represent the target upper boundary position and the target lower boundary position of the main image area. For example, Figure 4 As shown, Figure 2 A coordinate system is established with the upper left vertex of as the origin. The target upper boundary position of the main screen area can be represented as y1, and the target lower boundary position can be represented as y2.

[0049] In practical applications, for example, an artificial intelligence model can be trained to use the trained artificial intelligence model to determine the target boundary information of the main picture area of the target video based on multiple video frames extracted from the target video.

[0050] Considering that the development cost of using artificial intelligence models to determine target boundary information is high and the overall implementation efficiency may be low, the embodiments of the present disclosure provide a simpler and more efficient method for determining target boundary information. Specifically, the above-mentioned method of determining the target boundary information of the main screen area in the target video based on multiple video frames includes:

[0051] Step S121: for any video frame among the multiple video frames, by comparing pixel value differences between adjacent pixels in each column of the video frame, obtaining an upper boundary position and a lower boundary position of each column, where the upper boundary position of each column includes the pixel position where the pixel value first suddenly changes when the pixel value differences of each column are compared in a top-down order, and the lower boundary position of each column includes the pixel position where the pixel value first suddenly changes when the pixel value differences of each column are compared in a bottom-up order;

[0052] Step S122, determining boundary information corresponding to the video frame based on the upper boundary position and the lower boundary position of each column in the video frame, where the boundary information includes the upper boundary position and the lower boundary position of the main image area of the video frame;

[0053] Step S123 : determining target boundary information according to boundary information corresponding to each video frame in the plurality of video frames.

[0054] The pixel value may be, for example, an RGB color value, or the sum of RGB color values. Of course, a grayscale value, a brightness value, a contrast value, etc. may also be used, which is not limited in the embodiment of the present disclosure.

[0055] In step S121, the upper boundary position and the lower boundary position of each column are obtained by comparing the pixel value differences between adjacent pixels in each column in the video frame. For example, it may include: comparing (i.e., traversing) the pixel value differences between adjacent pixels in each column in a top-to-bottom order, obtaining the pixel position where the pixel value first mutates when the pixel value differences are compared in each column in a top-to-bottom order as the upper boundary position of each column, and comparing the pixel value differences between adjacent pixels in each column in a bottom-to-top order, obtaining the pixel position where the pixel value first mutates when the pixel value differences are compared in each column in a bottom-to-top order as the lower boundary position of each column.

[0056] The first sudden change in pixel value can be understood as the first time that the pixel values of two adjacent pixels are different, or the difference in pixel values is large (such as greater than a certain threshold). It should be understood that the frame area in the video usually has a large area of continuous identical pixel values. If the pixel values of adjacent pixels in a column suddenly change for the first time, it is likely that the change has occurred from the frame area to the main screen area. Therefore, by traversing the columns of the video frame and comparing the differences in the pixel values of adjacent pixels, the position where the pixel value first suddenly changes can be found, thereby determining the upper and lower boundary positions of each column.

[0057] Considering that the pixel values of some pixels adjacent to the frame area in the main image area may be the same as the pixel values of the pixels in the frame area, for example, some pixels adjacent to the frame area in the main image area are also black, or there may be some title texts of different colors in the frame area, such as the above Figure 2 and Figure 3 There is white title text in the black frame area, which will also cause a sudden change in pixel value. Therefore, in order to further accurately determine the upper and lower boundary positions of each column, the above method of finding the pixel position where the pixel mutation first occurs can be further improved. In one possible implementation, the above step S121 compares the pixel value differences between adjacent pixels in each column of the video frame to obtain the upper and lower boundary positions of each column, which may include:

[0058] Step S1211: For the i-th column in the video frame, compare the pixel value differences between adjacent pixels in the i-th column in a top-to-bottom order and a bottom-to-top order, respectively, to obtain the first pixel position in the i-th column where the pixel value first undergoes a sudden change when the pixel value differences are compared in the top-to-bottom order, and the second pixel position in the i-th column where the pixel value first undergoes a sudden change when the pixel value differences are compared in the bottom-to-top order, where i is a positive integer;

[0059] Step S1212: Starting from a row of pixels at a first pixel position, traverse upwards to determine whether pixel values of pixels in the same row are continuously the same, and starting from a row of pixels at a second pixel position, traverse downwards to determine whether pixel values of pixels in the same row are continuously the same, thereby obtaining a third pixel position at which pixel values of pixels in the same row are continuously the same for the first time when traversing upwards from the first pixel position, and a fourth pixel position at which pixel values of pixels in the same row are continuously the same for the first time when traversing downwards from the second pixel position;

[0060] Step S1213: determine the third pixel position as the upper boundary position of the i-th column, and determine the fourth pixel position as the lower boundary position of the i-th column.

[0061] It should be understood that if the width and height of the video frame are known, then the number of vertical columns of pixels and horizontal rows of pixels in the video frame, or the number of pixels in each column and each row of the video frame, is known.

[0062] For example, Figure 5 The implementation of the above steps S1211 to S1213 is described using an exemplary video frame as an example, wherein one grid represents one pixel point. Figure 5 There are 5 columns and 8 rows of pixels. For the first column b1 of the video frame, the pixel value difference between the second pixel in the first column and the first pixel can be compared in order from top to bottom, that is, whether the pixel values of the second pixel in the first column are the same as the first pixel or are less than the specified threshold. If they are the same or less than the specified threshold, it is considered that no pixel value mutation has occurred. Then compare the pixel value difference between the third pixel in the first column and the second pixel. If no pixel value mutation has occurred, then compare the pixel value difference between the fourth pixel in the first column and the third pixel. If the pixel values of the fourth pixel in the first column are different from the third pixel or are greater than the specified threshold, it is considered that the pixel value has mutated. At this time, the third pixel can be compared to the vertical pixel. The pixel position a3 in the vertical direction is determined as the first pixel position corresponding to the first column; at the same time, for the first column b1 of the video frame, the pixel value difference between the 7th pixel point and the 8th pixel point in the first column can be compared in order from bottom to top, that is, whether the pixel values of the 7th pixel point and the 8th pixel point in the first column are the same or are less than the specified threshold. If they are the same or less than the specified threshold, it is considered that no pixel value mutation has occurred. Then, the pixel value difference between the 6th pixel point and the 7th pixel point in the first column is compared. If the pixel values of the 6th pixel point and the 7th pixel point in the first column are different or greater than the specified threshold, it is considered that the pixel value mutation has occurred. At this time, the vertical pixel position a7 corresponding to the 7th pixel point can be determined as the second pixel position corresponding to the first column;

[0063] Then, for the first pixel position a3 corresponding to the first column, you can start from the row of pixels at the first pixel position a3 and traverse the same row of pixels upwards to see whether the pixel values are continuously the same. Specifically, you can first determine whether the pixel values of a row of pixels corresponding to the first pixel position a3 (i.e., (b1, a3), (b2, a3), (b3, a3), (b4, a3), (b5, a3)) are continuously the same in order from left to right. If the pixel values of (b1, a3) and (b2, a3) are the same, but the pixel values of (b2, a3) and (b3, a3) are different, it can be considered that the first pixel position a3 is the same. Assume that the pixel values of a row of pixels corresponding to a3 are not continuously the same, that is, there are pixels with sudden pixel value changes in row a3, then we can judge whether the pixel values of a row of pixels corresponding to a2 (that is, (b1, a2), (b2, a2), (b3, a2), (b4, a2), (b5, a2)) are continuously the same in order from left to right. If the pixel values of (b1, a2), (b2, a2), (b3, a2), (b4, a2), (b5, a2)) are continuously the same, then a2 can be determined as the third pixel position corresponding to the first column (that is, the upper boundary position of the first column); at the same time, for the For the second pixel position a7 corresponding to column 1, we can first determine whether the pixel values of a row of pixels corresponding to the second pixel position a7 (i.e., (b1, a7), (b2, a7), (b3, a7), (b4, a7), (b5, a7)) are continuously the same in order from left to right. If the pixel values of (b1, a7), (b2, a7), (b3, a7), (b4, a7), (b5, a7) are continuously the same, then a7 can be directly determined as the third pixel position corresponding to column 1 (i.e., the lower boundary position of column 1). If the pixel values of (b1, a7) and (b2, a7) are different, , it can be considered that the pixel values of a row of pixels corresponding to the second pixel position a7 are not continuously the same, and then the pixel values of a row of pixels corresponding to a8 (i.e. (b1, a8), (b2, a8), (b3, a8), (b4, a8), (b5, a8)) can be judged in order from left to right whether they are continuously the same. If they are continuously the same, a8 can be determined as the third pixel position corresponding to the first column (i.e. the lower boundary position of the first column). Of course, since a8 is actually the bottom boundary position, it can be considered that the first column has no bottom frame, and the bottom boundary position of the first column is also the lower boundary position of the first column.

[0064] Similarly, for columns 2 b2 through 5 b5 of the video frame, the upper and lower boundary positions corresponding to columns 2 b1 through 5 b5 can be determined using the same method for determining the lower and upper boundary positions of column 1. This method allows for more accurate determination of the upper and lower boundary positions of each column in the video frame.

[0065] In step S122, the boundary information corresponding to the video frame is determined based on the upper boundary position and the lower boundary position of each column in the video frame. For example, the average value of the multiple upper boundary positions corresponding to the multiple columns in the video frame can be used as the upper boundary position of the region of the main picture area of the video frame, and the average value of the multiple lower boundary positions corresponding to the multiple columns in the video frame can be used as the lower boundary position of the region of the main picture area of the video frame. In this way, the upper boundary position and the lower boundary position of the region can be randomly selected from all the columns of the video frame to calculate the upper boundary position and the lower boundary position of the region. Of course, the upper boundary position and the lower boundary position of all the columns of the video frame can also be directly used to calculate the upper boundary position and the lower boundary position of the region, and this embodiment of the present disclosure does not limit this.

[0066] Optionally, the pixel positions of the video frame from top to bottom can be set in ascending order. Thus, in the above step S122, the boundary information corresponding to the video frame is determined based on the upper boundary position and the lower boundary position of each column in the video frame. This can include: determining the maximum value of the multiple upper boundary positions corresponding to the multiple columns in the video frame as the regional upper boundary position of the main screen area of the video frame; and determining the minimum value of the multiple lower boundary positions corresponding to the multiple columns in the video frame as the regional lower boundary position of the main screen area of the video frame. In this way, the upper boundary position and the lower boundary position of the region can be randomly selected from all the columns of the video frame to determine the upper boundary position and the lower boundary position of the region; of course, the upper boundary position and the lower boundary position of the region can also be directly determined from the upper boundary position and the lower boundary position of all the columns of the video frame, and this embodiment of the present disclosure is not limited to this.

[0067] Alternatively, the pixel positions of the video frame from top to bottom may be set in descending order. Thus, when the pixel positions of the video frame from top to bottom are in descending order, in step S122, the boundary information corresponding to the video frame is determined based on the upper boundary position and the lower boundary position of each column in the video frame. This may include: determining the minimum value of the multiple upper boundary positions corresponding to the multiple columns in the video frame as the regional upper boundary position of the main picture area of the video frame; and determining the maximum value of the multiple lower boundary positions corresponding to the multiple columns in the video frame as the regional lower boundary position of the main picture area of the video frame.

[0068] Both methods aim to find the lowest upper boundary of each column and the highest lower boundary of each column. This way, we can avoid the possibility of misjudgment in each column, which may result in the content of the frame being divided into the main screen area. Figure 4In the scenario shown, because text exists in the upper and lower bounding boxes, a sudden pixel change may occur in the text area in the column where the text is located, causing the text location to be used as the upper or lower boundary. Taking the lowermost upper boundary of each column and the uppermost lower boundary of each column can effectively eliminate such misjudgments.

[0069] It should be understood that for each of the multiple video frames, the boundary information corresponding to each of the multiple video frames (i.e., the upper boundary position and the lower boundary position of the main screen area of each video frame) can be obtained according to the implementation method of the above-mentioned steps S121 to S122, that is, multiple boundary information can be obtained, and then in step S123, the average value of the upper boundary positions of the multiple boundary information can be determined as the target upper boundary position of the main screen area in the target video, and the average value of the lower boundary positions of the multiple boundary information can be determined as the target lower boundary position of the main screen area in the target video. In this way, the boundary coordinates of the continuous color area can be automatically detected and extracted from the video frame, which is particularly suitable for processing video frames containing frame areas, thereby realizing the extraction of a valid main screen area from the video frame.

[0070] Considering that a main image region (e.g., a dark sky image) in some video frames may have a large area with pixel values close to or identical to those of the frame region (e.g., a black frame), invalid boundary information may be obtained (i.e., some video frames may not detect valid boundary information). This invalid boundary information is usually outlier (i.e., it differs greatly from the boundary information of other video frames). Therefore, in order to improve the accuracy of the target boundary information, in one possible implementation, in the above step S123, determining the target boundary information based on the boundary information corresponding to each video frame in the multiple video frames may include:

[0071] filtering outlier boundary information from a plurality of boundary information corresponding to a plurality of video frames to obtain a plurality of valid boundary information from the plurality of boundary information;

[0072] Determine the average value of the upper boundary positions of the region in the multiple valid boundary information as the target upper boundary position of the main image region in the target video;

[0073] An average value of the lower boundary positions of the regions in the multiple valid boundary information is determined as the target lower boundary position of the main image region in the target video.

[0074] In practical applications, those skilled in the art may use known statistical methods, such as the Fibonacci method, to integrate the boundary information of all video frames and filter out outlier boundary information from multiple boundary information, where the outlier boundary information is also invalid boundary information. Of course, other outlier data filtering methods known in the art may also be used, as long as the outlier boundary information can be screened and filtered out, and the embodiments of the present disclosure are not limited thereto.

[0075] Among them, the boundary information remaining after filtering out the outlier boundary information from multiple boundary information is also the effective boundary information. Then, the target upper boundary position of the main picture area in the target video can be obtained by calculating the average value of the upper boundary positions of the region in the multiple effective boundary information, and the target lower boundary position of the main picture area in the target video can be obtained by calculating the average value of the lower boundary positions of the region in the multiple effective boundary information, thereby realizing the use of the effective boundary information of multiple video frames to obtain unified and accurate target boundary information representing the entire target video, which is helpful for subsequent cropping and other processing.

[0076] In step S13, each of the multiple video frames is cropped according to the target boundary information to obtain multiple subject area maps, where the subject area maps include the subject picture area in the video frame.

[0077] In practical applications, if the target boundary information is known, that is, the unified target upper boundary position and target lower boundary position of the main image area are known, then the image cropping technology known in the art can be used to crop each of the multiple video frames according to the target boundary information to obtain multiple main area maps, which is equivalent to cropping the main image area between the target upper boundary position and the target lower boundary position of each video frame to obtain the main area map. It should be understood that the embodiment of the present disclosure does not limit the cropping process of the video frame. For example, Figure 2 After cutting, you can get Figure 6a The main area diagram shown is Figure 3 After cutting, you can get Figure 6b Figure 2 shows the main area of the diagram.

[0078] In step S14, at least one dialogue set corresponding to at least one character in the target video is determined.

[0079] Among them, a single line set includes multiple lines associated with a single character in the context and the display position of each line. The multiple lines in a single line set are arranged according to the frame order of the video frame to which the main area map belongs.

[0080] In one possible implementation, the dialogue set may be obtained by performing text recognition on each of the plurality of subject area maps. Specifically, determining at least one dialogue set corresponding to at least one character in the target video may include:

[0081] Extracting line data from each of the plurality of main area maps, the line data including lines in the main area map and display positions of the lines;

[0082] Constructing a global line set based on the line data in each of the plurality of subject area maps, the global line set including all lines in the plurality of subject area maps arranged in a frame order of the video frames to which the subject area maps belong;

[0083] An artificial intelligence model is used to determine at least one line set corresponding to each of at least one character based on the global line set.

[0084] Among them, text recognition technology known in the art, such as optical character recognition (OCR) technology, can be used to extract the lines and the display positions of the lines in each main area diagram, and this embodiment of the present disclosure is not limited to this. In actual applications, before using OCR technology for text recognition, the main area diagram can be converted into a corresponding format (for example, it can be converted into Base64 encoding format) according to the format requirements of the OCR technology for input data before OCR recognition is performed, and this embodiment of the present disclosure is not limited to this.

[0085] Among them, the display position of the lines can be understood as the position where the lines are displayed in the main area diagram. For example, the display position of the lines can be expressed as the relative height of the lines area (that is, the display area of the lines) relative to the bottom boundary of the main area diagram (also called the lines height), or the display position of the lines can be expressed as the pixel distance from the highest point of the lines area to the bottom boundary of the main area diagram; of course, it can also be expressed as the pixel coordinates of the lines area in the main area diagram, etc., and this is not limited to this embodiment of the present disclosure.

[0086] Considering that in addition to the lines, the main area map may also contain some unfixed position of fancy characters. For example, some video screens may also display character profiles (such as names, titles, etc.) when a character first appears, scene introductions (such as location names) when a scene first appears, and may also display the title of the play, background introductions, inner monologues, etc. The display positions of these fancy characters are not fixed. However, the subtitles of the dialogues between characters are usually displayed in a designated area at the bottom of the main screen area. Therefore, in order to ensure that the text extracted from the main area map is the lines, it is also possible to first cut off the line area map of the default line area at the bottom of the main area map based on the default line height (i.e., the default height of the line display area relative to the bottom boundary), and perform text recognition on the cut-off line area map to obtain the line data in the main area map; of course, it is also possible to directly set the default line area at the bottom of the main area map to be recognized based on the default line height to obtain the line data in the main area map, and this embodiment of the present disclosure is not limited to this. It should be understood that the default line height can be a height preset based on experience, and the actual line height in the main area map may be less than or equal to the default line height.

[0087] In practical applications, after extracting the dialogue data from each subject area map, the dialogue extracted from multiple subject area maps can be arranged in frame order according to the frame order of the video frames to which the subject area maps belong in the target video to construct a global dialogue set. Considering that the dialogues in multiple subject area maps may be repeated, it is also possible to first remove the dialogues with repeated content and the subject area maps corresponding to the dialogues, and then arrange the remaining non-duplicate dialogues in frame order to obtain a global dialogue set. In this way, the artificial intelligence model can be provided with information on the order in which the dialogues appear, which helps the artificial intelligence model accurately divide the dialogues into at least one dialogue set corresponding to at least one character.

[0088] Among them, the artificial intelligence model can be implemented by using the text-based "speaker segmentation" or "speaker recognition" technology in related technologies. These technologies can divide the text into different speakers, i.e., "roles", according to the context. The embodiments of this disclosure do not limit the type, structure, training method, etc. of the artificial intelligence model, as long as it can achieve the required functions. For example, Figure 7 As shown, a global dialogue set can be extracted from the 8 main area maps. The lines in the global dialogue set can be filtered, grouped, sorted, and processed using an artificial intelligence model to obtain two dialogue sets for two characters (character A and character B).

[0089] It should be understood that the same character in the target video may have multiple lines of dialogue. For example, character A first says a line, then character B says a line, and then character A says another line. The two lines said by character A may not have a contextual relationship. Therefore, the artificial intelligence model can divide each character into at least one group of multiple lines with contextual association based on the contextual relationship between the lines. That is, each character can correspond to at least one set of lines. For example, the two lines of character A can be divided into two sets of lines.

[0090] In addition to identifying lines from the main area map, the required lines can also be extracted from an independent subtitle file. For example, as described above, the target video can be an original video clip of the original video of a film or TV series, and the original video is usually equipped with external subtitles. Therefore, at least one line set corresponding to at least one character in the target video can also be extracted from the subtitle file that matches the target video (i.e., the original video clip); it can be known that the subtitle file will contain information such as the line content, the character name of the line content, the display position of the line, and the display time. Therefore, based on the subtitle file, multiple lines with contextual associations for each character in the target video and the display position of each line can be extracted, and the frame order of the video frame to which each line belongs can also be known, thereby obtaining at least one line set for a single character. Of course, those skilled in the art can also use other known technical means in the art, such as speech recognition, to determine at least one line set corresponding to at least one character in the target video, and this is not limited to the embodiments of the present disclosure.

[0091] In step S15, at least one spliced image is generated based on at least one line set and the main area map corresponding to each line in each line set. A single spliced image includes the main screen area in the main area map corresponding to the first line in the single line set and the line areas in the main area map corresponding to the remaining lines except the first line.

[0092] As described above, the multiple lines in the line set are arranged according to the frame order of the video frames to which the main area maps belong. Therefore, the first line in the line set can be understood as the first line spoken by the character in the current dialogue. Then, the line area maps in the main area map to which the first line belongs can be spliced with the line area maps in the main area maps to which the remaining lines except the first line belong to obtain a spliced image. Specifically, the above-mentioned generation of at least one spliced image based on at least one line set and the main area map corresponding to each line in each line set may include:

[0093] For any line set, based on the display position of at least one line other than the first line in the line set, the line area graphs of the remaining lines in their respective main area graphs are cropped;

[0094] According to the frame order of the video frames to which the main area map corresponding to each line in the line set belongs, the main area map corresponding to the first line and the line area maps corresponding to the remaining lines are spliced to obtain a spliced image.

[0095] As mentioned above, the display position of the lines can be expressed as the line height (i.e., the pixel distance from the highest point of the line area to the bottom boundary of the main area diagram). Therefore, the line area diagram of each sentence in the main area diagram to which the remaining lines belong can be cropped according to the display position of the remaining lines of each sentence, and the line area diagram includes the line area. Then, according to the frame order of the video frame to which the main area diagram corresponding to each line in the line set belongs (i.e., the arrangement order of each line in the line set), the main area diagram corresponding to the first line and the line area diagram corresponding to the remaining lines can be spliced in sequence to obtain a spliced picture. For example, for Figure 7 The two lines sets divided in can be obtained respectively Figure 8a and Figure 8b Two stitched pictures are shown.

[0096] The “one line” of dialogue here may refer to the dialogue that appears in the main area diagram of the same video frame.

[0097] It is taken into account that the line set generated by the artificial intelligence model may also have a small number of lines or less line content (that is, the text length of the lines is short). For example, a line set may have two lines: "Yes" and "Let's go". The lines in this line set can express less semantic information and may not attract users to be interested in watching the drama, or it may not play a role in promoting the film and television drama. Therefore, it is also possible to select a line set with a number of lines exceeding a certain threshold and / or a line content exceeding a certain threshold from at least one line set generated in step S14 above, or it is also possible to select a line set containing the largest number of lines and / or the longest line content to generate a spliced image, and this is not limited in the embodiment of the present disclosure.

[0098] It should be understood that a line of dialogue may appear in multiple consecutive video frames, that is, the first line of dialogue may appear in at least two main area maps. At this time, a main area map can be randomly selected for splicing, or the facial area in each main area map to which the first line of dialogue belongs can be identified and the clarity of the facial area can be evaluated to obtain the facial clarity score in each main area map, and the main area map with the highest facial clarity score can be selected to splice the above-mentioned spliced image; for the main area maps corresponding to the remaining lines, since the line areas are cut out, the main area maps corresponding to the remaining lines can be randomly selected, or of course, selected according to the facial clarity score, and the embodiments of the present disclosure do not limit this.

[0099] Taking into account that different frame extraction strategies (frame extraction methods, number of frames, etc.) can be used when extracting multiple video frames from the target video, and a line may appear in multiple consecutive video frames, then multiple different spliced images can also be generated for the same set of lines (for example, if the main area map of the first line is different, different spliced images can be spliced out). Therefore, the user can use different frame extraction strategies to execute the image generation method of the embodiment of the present disclosure multiple times for the same target video according to actual needs (such as the number of spliced images to be generated), or can execute the image generation method of the embodiment of the present disclosure separately for multiple different target videos to achieve batch generation of spliced images. The embodiment of the present disclosure does not limit this.

[0100] In practical applications, the generated stitched images can be uploaded to a storage server (such as a server using Object Storage Service (OSS)) for storage, so that the stitched images can be used to create promotional materials on various social platforms.

[0101] According to the image generation method of the embodiment of the present disclosure, by extracting multiple video frames of the target video, identifying the target upper boundary position and the target lower boundary of the main screen area, cropping out the main area map containing the main screen area in the video frame, and then determining at least one line set corresponding to at least one character in the target video, and generating at least one spliced image according to the main screen area in the main area map corresponding to the first line of the lines in the line set and the line areas in the main area map corresponding to the remaining lines except the first line. This can realize automatic generation of context-related line spliced images divided by roles based on any target video, and realize low-cost and high-efficiency automatic batch production of high-quality spliced images. In an exemplary scenario, the spliced images can be used as promotional materials for film and television dramas on social platforms.

[0102] As described above, the image sources used for promotion on social media platforms include exquisite images of people. Therefore, an image composed of high-quality and clear images of the characters in the target video can also be generated. Therefore, in one possible implementation, the method can further include:

[0103] identifying a facial region in each of the plurality of subject region maps and evaluating the clarity of the facial region to obtain a facial clarity score for each subject region map;

[0104] Screening at least two target subject area maps from the multiple subject area maps according to the facial clarity score of each subject area map;

[0105] At least two target subject areas are horizontally or vertically spliced according to the aspect ratio of the subject area image to obtain a spliced image, where the aspect ratio is the ratio between the width and height of the subject area image.

[0106] In practical applications, face recognition technology known in the art can be used to identify the facial area in each of multiple subject area maps; and image clarity recognition technology known in the art can be used to evaluate the clarity of the facial area in each subject area map, which is not limited in the embodiments of the present disclosure.

[0107] The higher the facial clarity score of the character in the subject area map, the more exquisite the subject area map as a character image can be considered. Therefore, the subject area maps can be arranged from high to low according to the facial clarity score, and the n subject area maps with the top n facial clarity scores are selected as target subject area maps. These n target subject area maps can be horizontally or vertically spliced to obtain a spliced image. Here, n can be, for example, 2 or 4, which can achieve a two-part puzzle or a four-part puzzle, etc., and this is not limited in the embodiments of the present disclosure.

[0108] It should be understood that if the aspect ratio of the main area image is small, that is, the width of the main area image is smaller than the height, then the main area image is usually displayed as a vertical image. In this case, at least two target main area images can be horizontally spliced, for example, Figure 9a The stitched image is obtained by horizontal stitching; if the width-to-height ratio of the main area image is large, that is, the width of the main area image is greater than the height, then the main area image is usually displayed as a horizontal image. At this time, at least two target main area images can be stitched vertically, for example, Figure 9b A stitched picture obtained by longitudinal stitching is shown.

[0109] In actual applications, after screening out at least two target subject area maps from multiple subject area maps, the at least two target subject area maps can also be fed back to the user through the front-end interface. A puzzle tool can also be provided in the front-end interface. The user can customize the splicing of the target subject area maps based on the puzzle tool to obtain a spliced picture. This is not limited to the embodiments of the present disclosure.

[0110] In actual applications, the above-mentioned generated target subject area map or the spliced image formed by splicing the target subject area maps can be uploaded to a storage server (such as a server using Object Storage Service (OSS)) for storage, so as to facilitate the subsequent use of the spliced image or directly use the target subject area map to create promotional materials on various social platforms.

[0111] According to the embodiment of the present disclosure, a subject area map can be used to automatically generate a character splicing picture with high facial definition at low cost and high efficiency.

[0112] As described above, some target videos have a top frame. The top frame area usually contains some title text, which can also be used for promotional materials on social platforms. Therefore, in one possible implementation, when the target top boundary position is different from the top boundary of the video frame, the method may further include:

[0113] For any video frame, the title area in the video frame is cropped according to the target upper boundary position to obtain a title area map of the video frame;

[0114] Perform text recognition on the title area image to obtain the video title of the video frame; the video title is used to form promotional materials on social platforms with spliced images.

[0115] It should be understood that if the target upper boundary position of the main picture area is different from the top boundary of the video frame (which can be understood as the maximum upper boundary of the video frame), it can be considered that there is a top frame area (that is, the top title area) in the video frame. At this time, the title area from the target upper boundary position to the top boundary of the video frame can be cropped according to the target upper boundary position to obtain a title area map; then, text recognition technology known in the art, such as OCR technology, can be used to perform text recognition on the title area map to obtain the video title of the video frame, and this is not limited to this embodiment of the present disclosure.

[0116] In practical applications, the extracted video titles can be stored in a database as promotional materials, so that users can use the video titles and any of the spliced images to create promotional materials on social platforms.

[0117] Based on the above picture generation method provided by the embodiment of the present disclosure, the embodiment of the present disclosure also provides Figure 10 A schematic diagram of a picture generation process is shown, such as Figure 10As shown, the image generation process includes: obtaining a target video (i.e., a video file); extracting 10 video frames from the target video; determining the width and height information of the video frame (including the height and width of the video frame) and the target boundary information of the main screen area (the target upper boundary position y1 and the target lower boundary position y2); based on the height, y1, and y2 of the video frame, the main area map of the video frame can be intercepted, and facial recognition and clarity recognition are performed (i.e., the facial area in each of the multiple main area maps is identified and the clarity of the facial area is evaluated), and at least two target main area maps are screened out based on the facial clarity score, and the at least two target main area maps are further binary spliced or quartered, etc., to obtain a binary spliced image or a quartered spliced image. The images are stitched together and stored in the OSS server; and, based on y1 and y2, the dialogue area map at the bottom of the main screen area of the video frame can be captured, and the dialogue area map can be OCR-recognized to obtain the dialogue data in each main screen area; then the artificial intelligence model can be used to filter the dialogue context and divide it into at least one dialogue set according to the role, and the complete main area map of the first line of the dialogue set and the dialogue area maps of other lines can be selected to obtain the context stitching image (that is, the dialogue stitching image) and store it in the OSS server; the title area of the video frame can also be intercepted based on y1, and the title area map can be OCR-recognized to obtain the video title; the video title and dialogue data can be stored in the database for use as promotional materials.

[0118] According to the image generation method of the embodiment of the present disclosure, high-quality spliced images can be produced for use in TV series promotion and marketing, and can be automatically mass-produced to reduce labor costs; it can support secondary creation of TV series videos (i.e., generating dialogue puzzles, character puzzles, etc.); it can use the promotion and marketing needs of film and television content as the background to lower the threshold for creating promotional materials on social platforms and reduce production costs, and realize a high-quality image production process assisted by artificial intelligence models and image processing technology, which greatly shortens the time to produce images and is conducive to increasing the attractiveness of TV series promotion and marketing.

[0119] According to the image generation method of the embodiment of the present disclosure, it is possible to utilize video frame extraction and intelligent image analysis technology to efficiently and accurately extract the main image and line data, combine the artificial intelligence model to intelligently group the lines, and innovatively realize the mass production of line splicing pictures and character splicing pictures, greatly improving the efficiency of picture production; specifically, by adopting video frame extraction technology and using RGB values for boundary judgment, it is possible to more accurately judge whether the pixel points are continuous, thereby improving the efficiency of finding continuous pixel points and stably obtaining the effective main image area of the video frame; and, by extracting the line area and title area of the picture through the target boundary information of the main image area, using OCR technology to obtain the lines and titles, and performing face recognition and clarity recognition on the main area map, the image quality of the spliced picture can be improved. By using the artificial intelligence model to group the lines according to context and role, the line picture splicing effect can be achieved, which conforms to the splicing picture style of promotional materials in real scenes.

[0120] Figure 11 A block diagram of a picture generating device according to an embodiment of the present disclosure is shown as follows: Figure 11 As shown, the device includes:

[0121] The extraction module 111 is used to obtain a target video and extract multiple video frames from the target video;

[0122] A boundary determination module 112 is configured to determine target boundary information of a main picture area in the target video based on the multiple video frames, wherein the target boundary information includes a target upper boundary position and a target lower boundary position of the main picture area;

[0123] A cropping module 113 is configured to crop each of the plurality of video frames according to the target boundary information to obtain a plurality of subject area maps, each of the subject area maps including a subject picture area in the video frame;

[0124] A dialogue determination module 114 is configured to determine at least one dialogue set corresponding to at least one character in the target video; wherein each dialogue set includes multiple dialogues associated with a single character in a context and the display position of each dialogue, and the multiple dialogues in the single dialogue set are arranged in the order of the video frames to which the subject area map belongs;

[0125] The image generation module 115 is used to generate at least one spliced image based on the at least one line set and the main area map corresponding to each line in each line set, wherein a single spliced image includes the main screen area in the main area map corresponding to the first line in the single line set and the line areas in the main area map corresponding to the remaining lines except the first line.

[0126] In a possible implementation, the target boundary information of the main picture area in the target video is determined based on the multiple video frames, including: for any video frame among the multiple video frames, by comparing the pixel value differences between adjacent pixels in each column of the video frame, the upper boundary position and the lower boundary position of each column are obtained, the upper boundary position of each column includes the pixel position where the pixel value first mutates when the pixel value differences of each column are compared in a top-down order, and the lower boundary position of each column includes the pixel position where the pixel value first mutates when the pixel value differences of each column are compared in a bottom-up order; according to the upper boundary position and the lower boundary position of each column in the video frame, the boundary information corresponding to the video frame is determined, the boundary information including the upper boundary position and the lower boundary position of the area of the main picture area of the video frame; according to the boundary information corresponding to each video frame among the multiple video frames, the target boundary information is determined.

[0127] In a possible implementation, obtaining the upper boundary position and the lower boundary position of each column by comparing the pixel value differences of adjacent pixels in each column of the video frame includes: for the i-th column in the video frame, comparing the pixel value differences between adjacent pixels in the i-th column in the video frame in a top-to-bottom order and a bottom-to-top order, respectively, to obtain the first pixel position of the i-th column where the pixel value first suddenly changes when the pixel value differences are compared in a top-to-bottom order and the second pixel position where the pixel value first suddenly changes when the pixel value differences are compared in a bottom-to-top order; starting from a row of pixels at the first pixel position, Whether the pixel values of the pixels in the same row are continuously the same when traversing upward row by row from the first pixel position, and whether the pixel values of the pixels in the same row are continuously the same when traversing downward row by row from the second pixel position are continuously the same, obtain the third pixel position when the pixel values of the pixels in the same row are continuously the same for the first time when traversing upward from the first pixel position, and the fourth pixel position when the pixel values of the pixels in the same row are continuously the same for the first time when traversing downward from the second pixel position; determine the third pixel position as the upper boundary position of the i-th column, and determine the fourth pixel position as the lower boundary position of the i-th column.

[0128] In one possible implementation, the pixel positions of the video frame are in ascending order from top to bottom, and the boundary information corresponding to the video frame is determined based on the upper boundary position and the lower boundary position of each column in the video frame, including: determining the maximum value of multiple upper boundary positions corresponding to multiple columns in the video frame as the regional upper boundary position of the main picture area of the video frame; and determining the minimum value of multiple lower boundary positions corresponding to multiple columns in the video frame as the regional lower boundary position of the main picture area of the video frame.

[0129] In a possible implementation, the target boundary information is determined based on the boundary information corresponding to each video frame in the multiple video frames, including: filtering the outlier boundary information in the multiple boundary information corresponding to the multiple video frames to obtain multiple valid boundary information in the multiple boundary information; determining the average value of the upper boundary positions of the region in the multiple valid boundary information as the target upper boundary position of the main picture area in the target video; and determining the average value of the lower boundary positions of the region in the multiple valid boundary information as the target lower boundary position of the main picture area in the target video.

[0130] In one possible implementation, determining at least one line set corresponding to at least one character in the target video includes: extracting line data from each of the multiple main area maps, the line data including the lines in the main area map and the display position of the lines; constructing a global line set based on the line data in each of the multiple main area maps, the global line set including all lines in the multiple main area maps arranged in the frame order of the video frames to which the main area maps belong; and determining at least one line set corresponding to each of at least one character based on the global line set using an artificial intelligence model.

[0131] In one possible implementation, at least one spliced image is generated based on the at least one line set and the main area map corresponding to each line in each line set, including: for any line set, based on the display position of at least one remaining line in the line set except the first line, cropping the line area map of each remaining line in the main area map to which it belongs; and splicing the main area map corresponding to the first line with the line area maps corresponding to the remaining lines in accordance with the frame order of the video frames to which the main area map corresponding to each line in the line set belongs, to obtain a spliced image.

[0132] In one possible implementation, the device further includes: a scoring module for identifying a facial area in each of the multiple subject area maps and evaluating the clarity of the facial area to obtain a facial clarity score for each subject area map; a screening module for screening at least two target subject area maps from the multiple subject area maps based on the facial clarity score of each subject area map; and a splicing module for performing horizontal or vertical splicing on the at least two target subject areas based on the aspect ratio of the subject area maps to obtain a spliced image, where the aspect ratio is the ratio between the width and height of the subject area map.

[0133] In a possible implementation, when the target upper boundary position is different from the top boundary position of the video frame, the device further includes: a title cropping module, which is used to crop the title area in the video frame according to the target upper boundary position for any video frame to obtain a title area map of the video frame; a title recognition module, which is used to perform text recognition on the title area map to obtain a video title of the video frame; the video title is used to form promotional materials on a social platform together with the spliced image.

[0134] According to an embodiment of the present disclosure, by extracting multiple video frames of a target video, identifying the target upper boundary position and the target lower boundary of the main screen area, cropping a main area map containing the main screen area in the video frame, and then determining at least one line set corresponding to at least one character in the target video, and generating at least one spliced image according to the main screen area in the main area map corresponding to the first line of the lines in the line set and the line areas in the main area map corresponding to the remaining lines except the first line, it is possible to automatically generate context-related line splicing images divided by roles based on any target video, and realize low-cost and high-efficiency automatic batch production of high-quality spliced images. In an exemplary scenario, the spliced images can be used as promotional materials for film and television dramas on social platforms.

[0135] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0136] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0137] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0138] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0139] Figure 12 FIG1 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Figure 12 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0140] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0141] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0142] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0143] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0144] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0145] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0146] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0147] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0148] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0149] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0150] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating an image, characterized in that: include: Acquire a target video and extract multiple video frames from the target video; Determining target boundary information of a main picture area in the target video based on the multiple video frames, the target boundary information including a target upper boundary position and a target lower boundary position of the main picture area; cropping each of the plurality of video frames according to the target boundary information to obtain a plurality of subject area maps, wherein the subject area maps include subject picture areas in the video frames; Determining at least one dialogue set corresponding to at least one character in the target video; wherein a single dialogue set includes multiple dialogues associated with a single character in a context and a display position of each dialogue, and the multiple dialogues in the single dialogue set are arranged in a frame order of a video frame to which the subject area map belongs; generating at least one spliced image based on the at least one speech set and the main area map corresponding to each speech in each speech set, wherein the single spliced image includes the main screen area in the main area map corresponding to the first speech in the single speech set and the speech areas in the main area map corresponding to the remaining speech except the first speech; The target boundary information is obtained by comparing the pixel value differences between adjacent pixels in each column of the video frame to obtain the upper boundary position and the lower boundary position of each column, and is determined according to the upper boundary position and the lower boundary position of each column in the video frame; The step of obtaining the upper boundary position and the lower boundary position of each column by comparing the pixel value differences between adjacent pixels in each column of the video frame includes: For the i-th column in the video frame, comparing pixel value differences between adjacent pixels in the i-th column in a top-to-bottom order and a bottom-to-top order, respectively, to obtain a first pixel position in the i-th column where a sudden change in pixel value first occurs when the pixel value differences are compared in the top-to-bottom order, and a second pixel position in the i-th column where a sudden change first occurs when the pixel value differences are compared in the bottom-to-top order; Whether pixel values of pixels in the same row are continuously the same when traversing upward from a row of pixels at the first pixel position, and whether pixel values of pixels in the same row are continuously the same when traversing downward from a row of pixels at the second pixel position, are determined, to obtain a third pixel position at which pixel values of pixels in the same row are continuously the same for the first time when traversing upward from the first pixel position, and a fourth pixel position at which pixel values of pixels in the same row are continuously the same for the first time when traversing downward from the second pixel position; The third pixel position is determined as the upper boundary position of the i-th column, and the fourth pixel position is determined as the lower boundary position of the i-th column.

2. The method according to claim 1, characterized in that The determining target boundary information of the main screen area in the target video based on the multiple video frames includes: For any video frame among the multiple video frames, by comparing pixel value differences between adjacent pixels in each column of the video frame, an upper boundary position and a lower boundary position of each column are obtained, where the upper boundary position of each column includes a pixel position where a pixel value first undergoes a sudden change when pixel value differences are compared in a top-down order, and the lower boundary position of each column includes a pixel position where a pixel value first undergoes a sudden change when pixel value differences are compared in a bottom-up order; Determining boundary information corresponding to the video frame according to the upper boundary position and the lower boundary position of each column in the video frame, wherein the boundary information includes the upper boundary position and the lower boundary position of the main picture area of the video frame; The target boundary information is determined according to boundary information corresponding to each video frame in the multiple video frames.

3. The method according to claim 1 or 2, characterized in that The pixel positions of the video frame are in ascending order from top to bottom, and determining the boundary information corresponding to the video frame according to the upper boundary position and the lower boundary position of each column in the video frame includes: Determining the maximum value among a plurality of upper boundary positions corresponding to a plurality of columns in the video frame as the region upper boundary position of the main picture region of the video frame; The minimum value among a plurality of lower boundary positions corresponding to a plurality of columns in the video frame is determined as the region lower boundary position of the main picture region of the video frame.

4. The method according to claim 1 or 2, characterized in that The determining the target boundary information according to the boundary information corresponding to each video frame in the plurality of video frames includes: filtering outlier boundary information among the plurality of boundary information corresponding to the plurality of video frames to obtain a plurality of valid boundary information among the plurality of boundary information; Determine an average value of the upper boundary positions of the regions in the multiple valid boundary information as the target upper boundary position of the main image region in the target video; An average value of the lower boundary positions of the regions in the multiple valid boundary information is determined as the target lower boundary position of the main screen region in the target video.

5. The method according to claim 1, wherein The determining of at least one line set corresponding to at least one character in the target video includes: Extracting line data from each of the plurality of main body area maps, the line data including lines in the main body area map and display positions of the lines; constructing a global line set based on the line data in each of the plurality of subject area maps, the global line set including all lines in the plurality of subject area maps arranged in a frame order of the video frames to which the subject area maps belong; An artificial intelligence model is used to determine at least one line set corresponding to each of at least one character based on the global line set.

6. The method according to claim 1, characterized in that Generating at least one spliced image based on the at least one line set and the main area map corresponding to each line in each line set includes: For any line set, based on the display position of at least one line other than the first line in the line set, the line area graphs of the remaining lines in their respective main area graphs are cropped; According to the frame order of the video frames to which the main area map corresponding to each line in the line set belongs, the main area map corresponding to the first line and the line area maps corresponding to the remaining lines are spliced to obtain a spliced image.

7. The method according to claim 1, characterized in that The method further comprises: identifying a facial region in each of the plurality of subject region maps and evaluating clarity of the facial region to obtain a facial clarity score for each subject region map; screening at least two target subject area maps from the plurality of subject area maps according to the facial clarity score of each subject area map; The at least two target main area images are horizontally or vertically spliced to obtain a spliced image according to the aspect ratio of the main area image, where the aspect ratio is the ratio between the width and the height of the main area image.

8. The method according to claim 1, characterized in that In a case where the target upper boundary position is different from the top boundary position of the video frame, the method further includes: For any video frame, the title area in the video frame is cropped according to the target upper boundary position to obtain a title area map of the video frame; Text recognition is performed on the title area image to obtain a video title of the video frame; the video title is used to form promotional materials on a social platform together with the spliced image.

9. A picture generating device, characterized in that: include: An extraction module, configured to obtain a target video and extract multiple video frames from the target video; a boundary determination module, configured to determine target boundary information of a main picture area in the target video based on the multiple video frames, wherein the target boundary information includes a target upper boundary position and a target lower boundary position of the main picture area; a cropping module, configured to crop each of the plurality of video frames according to the target boundary information to obtain a plurality of subject area maps, wherein the subject area maps include a subject picture area in the video frame; a dialogue determination module, configured to determine at least one dialogue set corresponding to at least one character in the target video; wherein a single dialogue set includes multiple dialogues associated with a single character in a context and a display position of each dialogue, and the multiple dialogues in the single dialogue set are arranged in a frame order according to the video frame to which the subject area map belongs; An image generation module is configured to generate at least one spliced image based on the at least one line set and the main area map corresponding to each line in each line set, wherein the single spliced image includes a main screen area in the main area map corresponding to a first line in the single line set and line areas in the main area map corresponding to remaining lines except the first line; The target boundary information is obtained by comparing the pixel value differences between adjacent pixels in each column of the video frame to obtain the upper boundary position and the lower boundary position of each column, and is determined according to the upper boundary position and the lower boundary position of each column in the video frame; The step of obtaining the upper boundary position and the lower boundary position of each column by comparing the pixel value differences between adjacent pixels in each column of the video frame includes: For the i-th column in the video frame, comparing pixel value differences between adjacent pixels in the i-th column in a top-to-bottom order and a bottom-to-top order, respectively, to obtain a first pixel position in the i-th column where a sudden change in pixel value first occurs when the pixel value differences are compared in the top-to-bottom order, and a second pixel position in the i-th column where a sudden change first occurs when the pixel value differences are compared in the bottom-to-top order; Whether pixel values of pixels in the same row are continuously the same when traversing upward from a row of pixels at the first pixel position, and whether pixel values of pixels in the same row are continuously the same when traversing downward from a row of pixels at the second pixel position, are determined, to obtain a third pixel position at which pixel values of pixels in the same row are continuously the same for the first time when traversing upward from the first pixel position, and a fourth pixel position at which pixel values of pixels in the same row are continuously the same for the first time when traversing downward from the second pixel position; The third pixel position is determined as the upper boundary position of the i-th column, and the fourth pixel position is determined as the lower boundary position of the i-th column.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 8 when executing the instructions stored in the memory.

11. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images