A method, device, and storage medium for generating a sand drawing video

Abstract

The application discloses a sand drawing video generation method, device and equipment and a storage medium. The method comprises the following steps: for each to-be-displayed picture in a picture sequence, analyzing to obtain position information and category information of each object; sand drawing the to-be-displayed picture to calculate sand drawing pixel values corresponding to each pixel point in the sand drawn to-be-displayed picture; according to the category information of each object, extracting object pictures with the same category as the category of each object from a preset material library; obtaining sand drawing pixel values of pixel points corresponding to each object picture; generating sand drawing sub-videos corresponding to each to-be-displayed picture according to the sand drawing pixel values of the pixel points corresponding to each object picture; and sequentially splicing the sand drawing sub-videos of all the to-be-displayed pictures according to the sorting order of each to-be-displayed picture in the picture sequence to generate a sand drawing video. The application can improve the generation efficiency of the sand drawing video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of AI, and particularly relates to a sand drawing video generation method and device, equipment and a storage medium. BACKGROUND

[0002] The existing sand drawing video is mainly obtained by manual drawing. Generally, the manual drawing process of a single picture is videoized according to the drawing order of each object in the single picture, and then the drawing videos of multiple pictures are effectively spliced to form a sand drawing video. That is, the sand drawing video is currently obtained by manual drawing, and there is no corresponding scheme to automatically generate a sand drawing video from multiple pictures. The sand drawing video obtained by manual drawing is time-consuming and cannot be generated on a large scale, and the generation efficiency of the sand drawing video is low. Therefore, there is an urgent need for a sand drawing video generation method that can automatically generate a sand drawing video from multiple pictures and improve the generation efficiency of the sand drawing video. SUMMARY

[0003] The present application provides a sand drawing video generation method, device, equipment and storage medium to solve the technical problem that the existing manual drawing method for obtaining a sand drawing video is time-consuming and cannot generate a sand drawing video on a large scale, and the generation efficiency of the sand drawing video is low.

[0004] To solve the above technical problem, the present application provides a sand drawing video generation method, comprising:

[0005] obtaining a picture sequence;

[0006] For each to-be-displayed picture in the picture sequence, extracting the pixel point attribute information of each object in the to-be-displayed picture;

[0007] According to the pixel point attribute information, the position information and the category information of each object are analyzed and obtained;

[0008] sand drawing the to-be-displayed picture to calculate the sand drawing pixel value corresponding to each pixel point in the sand drawing to-be-displayed picture;

[0009] According to the category information of each object, the object pictures with the same category as each object are extracted from the preset material library;

[0010] According to the position information of each object and the sand drawing pixel value corresponding to each pixel point in the to-be-displayed picture, the sand drawing pixel value of the pixel point corresponding to each object picture is obtained;

[0011] According to the sand drawing pixel value of the pixel point corresponding to each object picture, the sand drawing sub-video corresponding to each to-be-displayed picture is generated;

[0012] The sand painting sub-videos of all the pictures to be displayed are spliced in sequence according to the order of the pictures to be displayed in the picture sequence, to generate a sand painting video.

[0013] As a preferred solution, the generation of the sand painting sub-video corresponding to each picture to be displayed according to the sand painting pixel values of the pixel points corresponding to the object pictures includes:

[0014] For each picture to be displayed, it is determined whether the picture to be displayed is the first picture in the picture sequence.

[0015] If yes, the object pictures are displayed in sequence according to the sand painting pixel values of the pixel points corresponding to the object pictures, to generate the sand painting sub-video corresponding to the picture to be displayed.

[0016] If no, the pixel point attribute information of each object of the previous picture to be displayed is obtained, the picture similarity of each object in the two pictures to be displayed is calculated in sequence, and the picture similarity is compared with a preset threshold value. When it is detected that the picture similarity is greater than the preset threshold value, the corresponding object in the previous picture to be displayed is regarded as a similar object, an initial background of the sand painting sub-video corresponding to the current picture to be displayed is generated according to the sand painting pixel values of the pixel points of the similar object, and then the object pictures are displayed in sequence on the initial background according to the sand painting pixel values of the pixel points corresponding to the object pictures, to generate the sand painting sub-video corresponding to the picture to be displayed.

[0017] As a preferred solution, the display of the object pictures in sequence on the initial background according to the sand painting pixel values of the pixel points corresponding to the object pictures of the current picture to be displayed includes:

[0018] The similar object in the previous picture to be displayed is taken as the initial background of the sand painting sub-video of the current picture to be displayed, and the pixel values of the similar object are modified according to the sand painting pixel values of the pixel points of the corresponding object in the current picture to be displayed.

[0019] The remaining object pictures are displayed in sequence according to the sand painting pixel values of the pixel points corresponding to the remaining object pictures.

[0020] As a preferred solution, the extraction of the pixel point attribute information of each object in the picture to be displayed includes:

[0021] The picture to be displayed is panoramic segmented according to a panoramic segmentation algorithm, to obtain each object in the picture to be displayed, and then the pixel point attribute information corresponding to each object is extracted according to the objects.

[0022] As a preferred solution, the panoramic segmentation algorithm comprises a semantic segmentation algorithm and a dense bounding box prediction algorithm.

[0023] The to-be-displayed picture is subjected to panoramic segmentation according to the panoramic segmentation algorithm, to obtain each object in the to-be-displayed picture, and then pixel point attribute information corresponding to each object is extracted according to the objects.

[0024] The semantic category probability of each pixel point in the to-be-displayed picture is predicted according to the semantic segmentation algorithm.

[0025] The bounding box set of each pixel point in the to-be-displayed picture is predicted according to the dense bounding box prediction algorithm.

[0026] Each object in the to-be-displayed picture is obtained according to the semantic category probability and the bounding box set of each pixel point in the to-be-displayed picture, and then pixel point attribute information corresponding to each object is extracted according to the objects.

[0027] As a preferred solution, the to-be-displayed picture is subjected to sand painting, comprising:

[0028] The to-be-displayed picture is input into a preset style transfer model, so that the to-be-displayed picture is subjected to sand painting by the style transfer model, and a sand painting picture of the to-be-displayed picture is output.

[0029] The style transfer model is formed by model training of a preset neural network model based on a plurality of sand painting style pictures.

[0030] On the basis of the above-mentioned embodiments, another embodiment of the present application provides a sand painting video generation device, characterized in that it comprises a picture sequence acquisition module, a pixel point information extraction module, an object information analysis module, a picture sand painting module, an object picture extraction module, a sand painting pixel value acquisition module, a sand painting sub-video generation module, and a sand painting video generation module.

[0031] The picture sequence acquisition module is configured to acquire a picture sequence.

[0032] The pixel point information extraction module is configured to extract pixel point attribute information of each object in each to-be-displayed picture in the picture sequence.

[0033] The object information analysis module is configured to analyze the position information and the category information of each object according to the pixel point attribute information.

[0034] The picture sand painting module is configured to subject the to-be-displayed picture to sand painting, and to calculate sand painting pixel values corresponding to each pixel point in the sand painted to-be-displayed picture.

[0035] The object picture extraction module is configured to extract object pictures of the same category as the categories of the objects from a preset material library according to the category information of the objects;

[0036] The sand picture pixel value acquisition module is configured to acquire sand picture pixel values of the pixel points corresponding to the object pictures according to the position information of the objects and the sand picture pixel values corresponding to the pixel points in the pictures to be displayed;

[0037] The sand picture sub-video generation module is configured to generate sand picture sub-videos corresponding to the pictures to be displayed according to the sand picture pixel values of the pixel points corresponding to the object pictures;

[0038] The sand picture video generation module is configured to sequentially splice the sand picture sub-videos of all the pictures to be displayed according to the sorting order of the pictures to be displayed in the picture sequence to generate a sand picture video.

[0039] On the basis of the above-mentioned embodiments, a further embodiment of the present application provides a sand picture video generation device, which comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the sand picture video generation method of the above-mentioned embodiments of the present application when executing the computer program.

[0040] On the basis of the above-mentioned embodiments, a further embodiment of the present application provides a storage medium, which comprises a stored computer program, wherein the computer program controls a device where the computer readable storage medium is located to execute the sand picture video generation method of the above-mentioned embodiments of the present application when running.

[0041] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0042] Compared with the sand drawing video obtained by the manual drawing method in the prior art, in the present application, for each to-be-displayed picture in a picture sequence, the pixel point attribute information of each object in the to-be-displayed picture is extracted; the position information and the category information of each object are analyzed according to the pixel point attribute information; the to-be-displayed picture is sand drawing, and the sand drawing pixel value corresponding to each pixel point in the sand drawing to-be-displayed picture is calculated; according to the category information of each object, the object pictures of the same category as each object are extracted from a preset material library; according to the position information of each object and the sand drawing pixel value corresponding to each pixel point in the to-be-displayed picture, the sand drawing pixel value of the pixel point corresponding to each object picture is obtained; according to the sand drawing pixel value of the pixel point corresponding to each object picture, the sand drawing sub-video corresponding to each to-be-displayed picture is generated; and the sand drawing sub-videos of all to-be-displayed pictures are spliced in turn according to the sorting order of each to-be-displayed picture in the picture sequence, to generate a sand drawing video. Through the present application, the sand drawing video can be automatically generated according to the picture sequence of the to-be-generated sand drawing video, and the generation efficiency of the sand drawing video is improved. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of a sand drawing video generation method provided by an embodiment of the present application;

[0044] Figure 2 is an input schematic diagram of panoramic segmentation provided by an embodiment of the present application;

[0045] Figure 3 is an output schematic diagram of panoramic segmentation provided by an embodiment of the present application

[0046] Figure 4 is an output schematic diagram of a style transfer moving cartoon model provided by an embodiment of the present application;

[0047] Figure 5 is a schematic diagram of a style transfer algorithm provided by an embodiment of the present application;

[0048] Figure 6 is a structural schematic diagram of a sand drawing video generation device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0050] Embodiment one

[0051] Referring to Figure 1 , a flowchart of a method for generating a sand drawing video is provided in an embodiment of the present application, including the following specific steps:

[0052] S1, obtaining a picture sequence;

[0053] Obtain a picture sequence to be generated for a sand drawing video, and automatically generate the sand drawing video to be generated according to the picture sequence.

[0054] S2, for each to-be-displayed picture in the picture sequence, extracting the pixel point attribute information of each object in the to-be-displayed picture;

[0055] Preferably, the extraction of the pixel point attribute information of each object in the to-be-displayed picture includes: performing panoramic segmentation on the to-be-displayed picture according to a panoramic segmentation algorithm to obtain each object in the to-be-displayed picture, and then extracting the pixel point attribute information corresponding to each object according to the objects.

[0056] Preferably, the panoramic segmentation algorithm includes a semantic segmentation algorithm and a dense bounding box prediction algorithm; the panoramic segmentation of the to-be-displayed picture according to the panoramic segmentation algorithm obtains each object in the to-be-displayed picture, and then the extraction of the pixel point attribute information corresponding to each object according to the objects includes: predicting the semantic category probability of each pixel point in the to-be-displayed picture according to the semantic segmentation algorithm; predicting the bounding box set of each pixel point in the to-be-displayed picture according to the dense bounding box prediction algorithm; obtaining each object in the to-be-displayed picture according to the semantic category probability and the bounding box set of each pixel point in the to-be-displayed picture, and then extracting the pixel point attribute information corresponding to each object according to the objects.

[0057] For each to-be-displayed picture in the picture sequence, first, the to-be-displayed picture is segmented according to a panoramic segmentation algorithm to segment each object in the to-be-displayed picture, and then the pixel point attribute information corresponding to each object is extracted. In a specific embodiment, the panoramic segmentation algorithm is described. Please refer to Figure 2 , the input schematic diagram of panoramic segmentation provided in an embodiment of the present application, the input Figure 2 is segmented by using a panoramic segmentation algorithm; please refer to Figure 3 , the output schematic diagram of panoramic segmentation provided in an embodiment of the present application, after the Figure 2 is segmented by using a panoramic segmentation algorithm, each object in Figure 2 is segmented to obtain Figure 3 .

[0058] Wherein, the specific process of using the panoramic segmentation algorithm to perform panoramic segmentation on the picture is as follows:

[0059] Assuming that a given predetermined set of C semantic classes is encoded as: C = {0, 1, …, C-1}, the panoramic segmentation algorithm needs to map each pixel point (x, y) of the image to a pair P(x, y) = (c, k) e L x N, where c represents the semantic class of the pixel point (x, y), and k represents the instance id, grouping pixels of the same class into different segments.

[0060] The panoramic segmentation algorithm includes a semantic segmentation branch and a dense bounding box prediction branch. In the semantic segmentation branch, the semantic class probability p(x, y, c) of each pixel point is predicted, where p represents the probability that the pixel point (x, y) belongs to the semantic class c. In the dense bounding box prediction branch, a bounding box set B(x, y) = (x min ,y min ,x max ,y max ,c) is predicted for each pixel point, where (x min ,y min ) represents the left upper corner coordinates of the bounding box, (x max ,y max ) represents the right lower corner coordinates of the bounding box, and c is the predicted class ID of the corresponding bounding box.

[0061] According to the results of the semantic segmentation branch and the dense bounding box prediction branch, the final panoramic segmentation result can be generated. First, the results of the dense bounding box prediction branch are simplified using a non-maximum suppression algorithm to obtain a new candidate bounding box set where K represents the number of bounding box predictions after NMS; then a global mask probability map is constructed: for each candidate bounding box B(x, y, i), the corresponding value of the probability map is:

[0062] M(x, y, i) = p(x, y, c i ) x IOU(B(x, y, i), B(x, y))

[0063] According to the global mask probability map, a threshold is set, and the global segmentation result can be obtained, segmenting each object in the image.

[0064] S3, according to the pixel point attribute information, analyzing to obtain the position information and class information of each object.

[0065] After obtaining the pixel point attribute information of each object in the to-be-displayed picture using the panoramic segmentation algorithm, the pixel point attribute information of each object is analyzed, the position distribution, area proportion, distance from the center point, and position relationship between each object of each class object in the picture are analyzed, and then the position information and class information of each object are analyzed.

[0066] S4, sand drawing the to-be-displayed picture to obtain a sand drawing pixel value corresponding to each pixel point in the sand drawing to-be-displayed picture.

[0067] Preferably, the sand drawing the to-be-displayed picture comprises: inputting the to-be-displayed picture into a preset style transfer model to make the style transfer model sand draw the to-be-displayed picture, and outputting a sand drawing picture of the to-be-displayed picture; wherein the style transfer model is formed by model training of a preset neural network model based on a plurality of sand drawing style pictures.

[0068] For each to-be-displayed picture in the picture sequence, the to-be-displayed pictures are simultaneously input into a preset style transfer model to make the style transfer model sand draw the to-be-displayed picture, and output a sand drawing picture of the to-be-displayed picture. In a specific embodiment, please refer to Figure 4 The output schematic diagram of the style transfer cartoonization model provided by an embodiment of the present application can use cartoonization to display picture sand drawing, input the picture into a preset cartoon style transfer model to make the style transfer model cartoonize the to-be-displayed picture, and obtain a cartoonized picture as shown in Figure 4 , to represent the effect of picture sand drawing.

[0069] Wherein, the basic idea of style transfer is that the covariance matrix of the feature map obtained after the convolution layer can well represent the texture features of the image, but the position information will be lost. However, in the task of style transfer, we can ignore the loss of position information, and only need to find a method to represent the texture information of the image, and then transfer the texture information to the image to be transferred, so as to complete the task of style transfer. The covariance matrix can be replaced by defining Gram matrix:

[0070]

[0071] Wherein, l represents the layer of the neural network, represents the feature of the i-th convolution kernel at the j-th position.

[0072] Please refer to Figure 5 The schematic diagram of the style transfer algorithm provided by an embodiment of the present application is as shown in Figure 5 The low-level features pay more attention to pixel information (a, b, c) and can easily restore the original pixel; and the high-level features (d, e) pay more attention to content and position, but are not very limited to the exact pixel value.

[0073] In order to transfer the style of the artwork a to the photo p, a new image is synthesized while matching the content representation of p and the style representation of f, which requires jointly minimizing the distance between the feature representation of the white noise image and the content representation of the photo at one layer and the painting style representation defined on several layers of the convolutional neural network. The loss function we minimize is:

[0074] L(a,p,f)=α×L style (p,f)+β×L content (a,f)

[0075] The loss function consists of two parts: the content loss L content and the style loss L style , where alpha and beta are the weighting coefficients of the content and style reconstruction, respectively.

[0076] The style loss is:

[0077]

[0078] Where F i,j represents the feature map obtained by passing the content image through a certain layer of convolutional layer, and P i,j represents the feature map generated by passing the generated image through the convolutional layer.

[0079] The content loss is:

[0080]

[0081] Where N l represents the number of feature maps, M l represents the size of the feature map, and respectively represent the Gram matrix obtained by self-correlation of the feature map obtained by passing the generated image and the style image through the convolutional layer, and the square error can be used as the loss function to make them close.

[0082] Based on the style transfer algorithm, a number of corresponding sand painting style pictures can be selected to train the preset neural network model to generate a style transfer model, which can convert the input picture into a sand painting style, i.e. picture sand painting.

[0083] S5, according to the category information of each object, extract the object picture with the same category as the object from the preset material library.

[0084] Because the object pictures drawn in the sand drawing video are all abstract pictures, before generating the sand drawing video, the object pictures of the same category in the to-be-displayed pictures can be replaced by the preset abstract object pictures in the material library, for example, when the tree is segmented after panoramic segmentation in the to-be-displayed pictures, the tree in the to-be-displayed pictures can be replaced by the abstract tree in the material library, so as to improve the visual display effect of the finally generated sand drawing video.

[0085] S6, obtaining the sand drawing pixel values of the pixel points corresponding to the object pictures according to the position information of the objects and the sand drawing pixel values corresponding to the pixel points in the to-be-displayed pictures.

[0086] According to the positions of the objects in the to-be-displayed pictures and the sand drawing pixel values corresponding to the pixel points in the to-be-displayed pictures, the sand drawing pixel values of the pixel points corresponding to the objects are obtained. When displaying, each object in the display picture needs to be replaced by the extracted object picture in the material library, that is, the object picture is displayed according to the sand drawing pixel values of the pixel points corresponding to the replaced object.

[0087] S7, generating the sand drawing sub-video corresponding to each to-be-displayed picture according to the sand drawing pixel values of the pixel points corresponding to the object pictures.

[0088] Preferably, the generating the sand drawing sub-video corresponding to each to-be-displayed picture according to the sand drawing pixel values of the pixel points corresponding to the object pictures comprises: for each to-be-displayed picture, judging whether the to-be-displayed picture is the first picture in the picture sequence; if yes, displaying the object pictures in sequence according to the sand drawing pixel values of the pixel points corresponding to the object pictures of the to-be-displayed picture, to generate the sand drawing sub-video corresponding to the to-be-displayed picture; if no, obtaining the pixel point attribute information of each object of the previous to-be-displayed picture of the current to-be-displayed picture, calculating the picture similarity of each object in the two to-be-displayed pictures in sequence, and comparing the picture similarity with a preset threshold value, when it is detected that the picture similarity is greater than the preset threshold value, the corresponding object in the previous to-be-displayed picture is regarded as a similar object, an initial background of the sand drawing sub-video corresponding to the current to-be-displayed picture is generated according to the sand drawing pixel values of the pixel points of the similar object, and then the object pictures are displayed in sequence on the initial background according to the sand drawing pixel values of the pixel points corresponding to the object pictures of the current to-be-displayed picture, to generate the sand drawing sub-video corresponding to the to-be-displayed picture.

[0089] Preferably, the sand painting pixel values of the pixel points corresponding to the object pictures of the current display picture are used to display the object pictures in the initial background in sequence, including: taking similar objects in a previous picture to be displayed as an initial background of a sand painting sub-video of the current picture to be displayed, and modifying pixel values of the similar objects according to sand painting pixel values of the pixel points corresponding to the objects in the current picture to be displayed; and displaying the remaining object pictures in sequence according to sand painting pixel values of the pixel points corresponding to the object pictures.

[0090] When generating a sand painting sub-video corresponding to each picture to be displayed, the object pictures are displayed in sequence according to sand painting pixel values of the pixel points corresponding to the object pictures in the picture to be displayed and drawing modes of the objects defined in the drawing module (for example, for an object, the drawing mode can be: first drawing an outer contour part of the object and then filling an inner part of the object, or first drawing a center point of the object and then drawing layer by layer towards the outside, etc.), to generate the sand painting sub-video corresponding to the picture to be displayed.

[0091] However, when manually drawing sand paintings, after drawing a picture, the relationship between the previous and next pictures is considered, part of the objects in the previous picture is retained as a starting background of the next picture, and the sand painting of the next picture is completed on the basis of the starting background, so as to realize effective splicing of the previous and next pictures.

[0092] In order to simulate the effective splicing process between the previous and next pictures, when generating a sand painting sub-video corresponding to each picture to be displayed, it is first determined whether the picture to be displayed is the first picture in a picture sequence; if yes, the object pictures are directly displayed in sequence according to sand painting pixel values of the pixel points corresponding to the object pictures in the picture to be displayed, to generate the sand painting sub-video corresponding to the picture to be displayed. If no, pixel point attribute information of the objects in a previous picture to be displayed is obtained, the picture similarity of the objects in the previous and next pictures to be displayed is calculated in sequence according to the pixel point attribute information of the objects in the previous and next pictures to be displayed, and the picture similarity is compared with a preset threshold value; when it is detected that the picture similarity is greater than the preset threshold value, the objects corresponding to the previous picture to be displayed are taken as similar objects, an initial background of a sand painting sub-video corresponding to the current picture to be displayed is generated according to sand painting pixel values of the pixel points of the similar objects, and pixel values of the similar objects are modified according to sand painting pixel values of the pixel points of the objects in the current picture to be displayed; and then the remaining object pictures are displayed in sequence according to sand painting pixel values of the pixel points corresponding to the object pictures, to generate the sand painting sub-video corresponding to the picture to be displayed.

[0093] If no picture similarity greater than the preset threshold is detected when the picture similarity is compared with the preset threshold, an initial background of the sand painting sub-video corresponding to the current to-be-displayed picture does not need to be generated, that is, when the sand painting sub-video is generated, the current to-be-displayed picture starts to be displayed on a whiteboard without a background object, and then the object pictures are displayed in sequence according to the sand painting pixel values of the pixels corresponding to the object pictures.

[0094] S8, the sand painting sub-videos of all the to-be-displayed pictures are spliced in sequence according to the sorting order of the to-be-displayed pictures in the picture sequence to generate a sand painting video.

[0095] After the sand painting sub-videos of all the to-be-displayed pictures in the picture sequence of the to-be-generated sand painting video are generated, the sand painting sub-videos are spliced in sequence according to the sorting order of the to-be-displayed pictures in the picture sequence, so that the sand painting video of the picture sequence is generated.

[0096] Therefore, the sand painting video generation method provided by the application can automatically generate a sand painting video according to a picture sequence of a to-be-generated sand painting video, and improves the generation efficiency of the sand painting video.

[0097] Embodiment Two

[0098] Please refer to Figure 6 A structure schematic diagram of a sand painting video generation device provided by an embodiment of the application is shown in the figure, and the device comprises a picture sequence acquisition module, a pixel point information extraction module, an object information analysis module, a picture sand painting module, an object picture extraction module, a sand painting pixel value acquisition module, a sand painting sub-video generation module, and a sand painting video generation module.

[0099] The picture sequence acquisition module is configured to acquire a picture sequence.

[0100] The pixel point information extraction module is configured to extract pixel point attribute information of each object in each to-be-displayed picture in the picture sequence.

[0101] The object information analysis module is configured to analyze the position information and the category information of each object according to the pixel point attribute information.

[0102] The picture sand painting module is configured to sand paint the to-be-displayed picture and calculate sand painting pixel values corresponding to each pixel in the sand painted to-be-displayed picture.

[0103] The object picture extraction module is configured to extract object pictures of the same category as each object in a preset material library according to the category information of each object.

[0104] The sand picture pixel value acquisition module is configured to acquire sand picture pixel values of the pixel points corresponding to the object pictures according to the position information of the objects and the sand picture pixel values corresponding to the pixel points in the pictures to be displayed.

[0105] The sand picture sub-video generation module is configured to generate sand picture sub-videos corresponding to the pictures to be displayed according to the sand picture pixel values of the pixel points corresponding to the object pictures.

[0106] The sand picture video generation module is configured to sequentially splice the sand picture sub-videos of all the pictures to be displayed according to the sorting order of the pictures to be displayed in the picture sequence to generate a sand picture video.

[0107] Embodiment Three

[0108] Correspondingly, the embodiment of the present application provides a sand picture video generation device, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the sand picture video generation method of the above-mentioned embodiment of the present application when executing the computer program.

[0109] Embodiment Four

[0110] Correspondingly, the embodiment of the present application provides a storage medium, which comprises a stored computer program, wherein the computer program controls the device where the computer readable storage medium is located to execute the sand picture video generation method of the above-mentioned embodiment of the present application when running.

[0111] In summary, the present application provides a sand picture video generation device, equipment and storage medium, and the sand picture video can be automatically generated according to the picture sequence of the sand picture video to be generated, and the generation efficiency of the sand picture video is improved.

[0112] It should be noted that the device embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment of the present application according to actual needs. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be realized as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity, the specific working process of the device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0114] The device can be a desktop computer, a notebook, a palm computer, a cloud server, and the like. The device can include, but is not limited to, a processor, a memory.

[0115] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The processor is a control center of the device, and connects all parts of the device through various interfaces and lines.

[0116] The memory can be used to store the computer program, and the processor realizes various functions of the device by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required by a function, and the like; and the data storage area can store data created according to the use of the mobile phone, and the like. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0117] The storage medium is a computer readable storage medium, and the computer program is stored in the computer readable storage medium. When the computer program is executed by a processor, steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0118] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A method for generating a sand drawing video, characterized by, The method comprises the following steps: acquiring a picture sequence; extracting pixel point attribute information of each object in each to-be-displayed picture in the picture sequence; analyzing the position information and the category information of each object according to the pixel point attribute information; sand painting the to-be-displayed picture to obtain sand painting pixel values corresponding to each pixel point in the sand painted to-be-displayed picture; extracting abstracted object pictures of the same category as the categories of the objects from a preset material library according to the category information of the objects; obtaining sand painting pixel values of the pixel points corresponding to the object pictures according to the position information of the objects and the sand painting pixel values of the pixel points corresponding to each pixel point in the to-be-displayed picture; generating sand painting sub-videos corresponding to the to-be-displayed pictures according to the sand painting pixel values of the pixel points corresponding to the object pictures; splicing the sand painting sub-videos of all the to-be-displayed pictures in sequence according to the order of the to-be-displayed pictures in the picture sequence to generate a sand painting video.

2. The method of claim 1, wherein, The method of generating sand painting sub-videos corresponding to the to-be-displayed pictures according to the sand painting pixel values of the pixel points corresponding to the object pictures comprises the following steps: determining whether the to-be-displayed picture is the first picture in the picture sequence; if yes, displaying the object pictures in sequence according to the sand painting pixel values of the pixel points corresponding to the object pictures to generate a sand painting sub-video corresponding to the to-be-displayed picture; if no, obtaining the pixel point attribute information of each object in the previous to-be-displayed picture of the current to-be-displayed picture, calculating the picture similarity of each object in the two to-be-displayed pictures in sequence, and comparing the picture similarity with a preset threshold value; when it is detected that the picture similarity is greater than the preset threshold value, regarding the corresponding object in the previous to-be-displayed picture as a similar object, generating an initial background of a sand painting sub-video corresponding to the current to-be-displayed picture according to the sand painting pixel values of the pixel points of the similar object, and then displaying the object pictures in sequence on the initial background according to the sand painting pixel values of the pixel points corresponding to the object pictures to generate a sand painting sub-video corresponding to the to-be-displayed picture.

3. The method of claim 2, wherein the sand video is generated by: The method of displaying the object pictures in sequence on the initial background according to the sand painting pixel values of the pixel points corresponding to the object pictures in the current to-be-displayed picture comprises the following steps: regarding the similar object in the previous to-be-displayed picture as the initial background of the sand painting sub-video of the current to-be-displayed picture, and modifying the pixel values of the similar object according to the sand painting pixel values of the pixel points of the corresponding object in the current to-be-displayed picture; displaying the remaining object pictures in sequence according to the sand painting pixel values of the pixel points corresponding to the remaining object pictures.

4. The method of claim 1, wherein the sand video is generated by: The method of extracting the pixel point attribute information of each object in the to-be-displayed picture comprises the following steps: performing panorama segmentation on the to-be-displayed picture according to a panorama segmentation algorithm to obtain each object in the to-be-displayed picture, and then extracting the pixel point attribute information corresponding to each object according to the objects.

5. The method of claim 4, wherein the sand video is generated by: The panoramic segmentation algorithm includes: a semantic segmentation algorithm and a dense bounding box prediction algorithm; The image to be displayed is segmented panoramically using a panoramic segmentation algorithm to obtain each object in the image. Then, pixel attribute information corresponding to each object is extracted, including: The semantic category probability of each pixel in the image to be displayed is predicted based on the semantic segmentation algorithm. The dense bounding box prediction algorithm is used to predict the bounding box set of each pixel in the image to be displayed. Based on the semantic category probability and bounding box set of each pixel in the image to be displayed, each object in the image to be displayed is obtained, and then the pixel attribute information corresponding to each object is extracted.

6. The method of claim 1, wherein, The step of converting the image to be displayed into a sand painting includes: The image to be displayed is input into a preset style transfer model so that the style transfer model can perform sand painting on the image to be displayed and output the sand painting image of the image to be displayed. The style transfer model is trained on a preset neural network model based on several sand painting style images.

7. An apparatus for generating a sand drawing video, characterized by comprising: include: The module includes: image sequence acquisition module, pixel information extraction module, object information analysis module, image sand painting module, object image extraction module, sand painting pixel value acquisition module, sand painting sub-video generation module, and sand painting video generation module. The image sequence acquisition module is used to acquire an image sequence; The pixel information extraction module is used to extract the pixel attribute information of each object in each image to be displayed in the image sequence. The object information analysis module is used to analyze and obtain the position information and category information of each object based on the pixel attribute information; The image sand art module is used to sand art the image to be displayed and calculate the sand art pixel value corresponding to each pixel in the sand art image to be displayed. The object image extraction module is used to extract abstract object images that are the same category as the objects from a preset material library based on the category information of each object. The sand art pixel value acquisition module is used to acquire the sand art pixel value of the pixel corresponding to each object image based on the position information of each object and the sand art pixel value corresponding to each pixel in the image to be displayed. The sand painting sub-video generation module is used to generate sand painting sub-videos corresponding to each image to be displayed based on the sand painting pixel values ​​of the pixels corresponding to each object image. The sand painting video generation module is used to sequentially splice together the sand painting sub-videos of all the images to be displayed according to the sorting order of each image in the image sequence to generate a sand painting video.

8. A sand drawing video generation device, characterized by comprising: The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method for generating sand art video as described in any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium comprises a stored computer program, wherein the computer program controls a device in which the computer readable storage medium is located to perform the sand painting video generation method of any one of claims 1 to 6 when the computer program is running.

Images