Automatic editing method and system for pet video

Abstract

The invention relates to an automatic editing method and system for a pet video, and belongs to the technical field of video editing, and the method comprises the steps: obtaining a target video stream; splitting each frame of image into a plurality of patches; inputting the patch of each frame of image to Transform for feature extraction, and obtaining an embedded value corresponding to the patch of each frame of image; inputting the embedded value corresponding to the patch of each frame of image into a recognition model, and outputting the recognized emotional state of the pet; screening out a target special effect matched with the recognized emotional state of the pet from a pre-constructed special effect library; and adding the target special effect into the target video stream. According to the method and the device, a fully automatic pet video editing mode can be provided for a pet owner, operation of the pet owner is not needed, and the edited video can better conform to the state of the pet, that is, the effect (special effect) presented by the video corresponds to the current emotion of the pet, so that the user experience is improved. Therefore, the attraction and interestingness of the automatically edited video are provided.

Description

Technical Field

[0001] The present invention relates to the technical field of video editing, and in particular to an automatic editing method and system for pet videos. Background Art

[0002] With the popularity of mobile devices and social platforms, users can more easily record and share interesting moments between them and their pets, which has prompted the demand for creating more attractive and professional pet videos.

[0003] Usually, pet owners can only post the pet videos they shoot directly to social platforms, or perform some simple operations such as cropping the picture, adding transitions, etc., which cannot meet the needs of non-professional users and makes the posted videos unattractive. Summary of the invention

[0004] In order to solve the above-mentioned problems in the prior art, the present invention provides a method and system for automatically editing pet videos.

[0005] In a first aspect, an embodiment of the present application provides an automatic editing method for pet videos, comprising: obtaining a target video stream; wherein the target video stream is a video stream containing a pet captured by a camera device, and the target video stream includes F frame images; F is a positive integer; splitting each frame image into multiple patches; inputting the patch of each frame image into a Transformer for feature extraction, and obtaining an embedding value corresponding to the patch of each frame image; wherein the embedding value is used to capture the position information and action information of the pet in the image; inputting the embedding value corresponding to the patch of each frame image into a recognition model, and outputting the emotional state of the identified pet; wherein the recognition model includes a multi-head attention module and a multi-layer perceptron arranged in sequence; screening out a target special effect that matches the emotional state of the identified pet from a pre-built special effects library; and adding the target special effect to the target video stream.

[0006] Optionally, the step of inputting the patch of each frame image into the Transformer for feature extraction to obtain the embedding value corresponding to the patch of each frame image includes: inputting the patch of each frame image into the Transformer for feature extraction to obtain the initial embedding value corresponding to the patch of each frame image; and based on the relative position relationship between the patches of each frame image, performing secondary embedding on the initial embedding value to obtain the embedding value corresponding to the patch of each frame image.

[0007] Optionally, the number of patches in each frame is ;in, Indicates the number of height directions, represents the number in the width direction; the initial embedding value is embedded again based on the relative position relationship between the patches of each frame image to obtain the embedding value corresponding to the patch of each frame image, including: based on the initial embedding value corresponding to the patch of each frame image, the number of rows is , the number of columns is Generate two grids respectively and splice them; the two grids are symmetrical; flatten the spliced ​​two grids to obtain a flattened one-dimensional tensor; flatten the one-dimensional tensor according to the number of rows , the number of columns is Reshape and add an empty dimension to the second and third dimensions respectively, thereby generating two different three-dimensional tensors; subtract the two different three-dimensional tensors to obtain a relative position table; wherein the relative position table corresponds to the relative position relationship between the patches of the frame image; based on the relative position table, perform secondary embedding on the initial embedding value to obtain the embedding value corresponding to the patch of each frame image.

[0008] Optionally, the method also includes: calculating the attention matrix of each frame image based on the embedding value corresponding to the patch of each frame image; counting the standard deviation value of the attention matrix; in response to the standard deviation value being greater than a threshold, determining that a pet exists in the frame image; or in response to the standard deviation value being less than or equal to the threshold, determining that a pet does not exist in the frame image.

[0009] Optionally, adding the target special effect to the target video stream includes: deleting image frames in which no pet exists from the target video stream to obtain an updated video stream; and adding the target special effect to the updated video stream.

[0010] Optionally, adding the target special effect to the target video stream includes: deleting image frames in which no pets exist from the target video stream; after deleting the image frames in which no pets exist from the target video stream, screening out image frames in which continuous emotional states are present, and adding the target special effect to the image frames in which continuous emotional states are present.

[0011] Optionally, the embedding values ​​corresponding to the patches of each frame of the image are input into the recognition model, and the recognized emotional state of the pet is output, including: inputting the embedding values ​​corresponding to the patches of each frame of the image into the recognition model, and outputting the recognized emotional state of the pet, and the pet's location information; wherein the pet's location information includes the coordinates of the center point of the pet, and the detection frame of the pet.

[0012] Optionally, after obtaining the embedding value corresponding to the patch of each frame image, the method also includes: generating an embedding sequence based on the embedding value corresponding to the patch of each frame image; wherein the first bit of the embedding sequence is a placeholder vector; the first bit of the placeholder vector is used to characterize whether a pet exists; the second bit of the placeholder vector is used to characterize the emotional state of the pet, the third and fourth bits of the placeholder vector are used to characterize the center point coordinates of the pet, and the fifth and sixth bits of the placeholder vector are used to characterize the detection frame of the pet.

[0013] Optionally, adding the target special effect to the target video stream includes: determining an adding position of the target special effect based on position information of the pet, and adding the target special effect to the target video stream according to the adding position.

[0014] In a second aspect, the present application provides an automatic editing system for pet videos, comprising: an acquisition module for acquiring a target video stream; wherein the target video stream is a video stream containing a pet collected by a camera device, and the target video stream includes F frame images; F is a positive integer; a position and action recognition module for splitting each frame image into multiple patches; inputting the patches of each frame image into a Transformer for feature extraction, and obtaining an embedded value corresponding to the patches of each frame image; wherein the embedded value is used to capture the position information and action information of the pet in the image; an emotion recognition module for inputting the embedded value corresponding to the patches of each frame image into a recognition model, and outputting the emotional state of the identified pet; wherein the recognition model includes a multi-head attention module and a multi-layer perceptron arranged in sequence; an automatic editing module for screening out target special effects that match the emotional state of the identified pet from a pre-built special effects library; and adding the target special effects to the target video stream.

[0015] The beneficial effects of the present invention include: first, the present application can sense the emotional state of the pet based on the pet's location information and action information, and can match the corresponding special effects based on the emotional state, and then the special effects can be merged with the video stream. Furthermore, a fully automated pet video editing method can be provided for pet owners, without the need for pet owners to operate, and at the same time, the edited video can be more in line with the pet's state, that is, the effect (special effect) presented in the video echoes the pet's current emotion, so as to provide the attractiveness and fun of the automatically edited video. Secondly, the implementation of the present application is to add special effects directly to the original video (i.e., the target video), which has better visual effects than the existing special effects that can only be added to the video transition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of the steps of an automatic editing method for pet videos provided by an embodiment of the present invention; Figure 2 A schematic diagram of a target video stream after adding special effects provided by an embodiment of the present invention; Figure 3 A module block diagram of an automatic pet video editing system provided by an embodiment of the present invention; Figure 4 A module block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0018] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0019] Usually, pet owners can only post the pet videos they shoot directly to social platforms, or perform some simple operations such as cropping the picture, adding transitions, etc., which cannot meet the needs of non-professional users and makes the posted videos unattractive.

[0020] In view of the above problems, the present application proposes the following embodiments to solve the above technical problems.

[0021] See also Figure 1 , an embodiment of the present application provides a method for automatically editing a pet video, including: steps 101 to 106.

[0022] Step 101: Obtain a target video stream.

[0023] The target video stream is a video stream containing a pet captured by a camera device, and the target video stream includes F frame images; F is a positive integer.

[0024] The camera device may be, but is not limited to, a home monitoring device, a mobile phone, a SLR camera, etc.

[0025] The target video stream may also include the height ( ),Width( )、Number of channels( ) and other parameters.

[0026] Step 102: Split each frame image into multiple patches.

[0027] It should be explained that in the embodiment of the present application, a patch can be understood as an image block, that is, each frame of an image is divided into a plurality of image blocks, and the size and number of the image blocks can be set according to requirements.

[0028] Step 103: Input the patch of each frame image into Transformer for feature extraction, and obtain the embedding value corresponding to the patch of each frame image.

[0029] The embedded value is used to capture the location information and action information of the pet in the image.

[0030] Step 104: Input the embedding value corresponding to the patch of each frame image into the recognition model, and output the recognized emotional state of the pet.

[0031] Among them, the recognition model includes a multi-head attention module and a multi-layer perceptron arranged in sequence.

[0032] Step 105: Filter out target special effects that match the recognized emotional state of the pet from a pre-built special effects library.

[0033] For example, the pre-built special effects library may store special effects such as flowers, cakes, fireworks, crying expressions, angry expressions, etc. Then, the corresponding target special effects are selected according to the recognized emotional state of the pet.

[0034] If the recognized emotional state of the pet is happy, the corresponding target special effect may be fireworks.

[0035] Step 106: Add the target special effect to the target video stream.

[0036] Finally, the target special effect is synthesized with the target video stream. Here, the corresponding target special effect may be synthesized on an image frame in which the emotional state of the pet is determined.

[0037] like Figure 2 As shown, the target feature is fireworks. After recognizing that the pet's emotional state is happy, a fireworks special effect is added to the video frame.

[0038] In summary, the automatic pet video editing method provided in the embodiment of the present application has the following beneficial effects: First, the application can sense the emotional state of the pet based on the pet's location information and action information, and can match the corresponding special effects based on the emotional state, and then merge the special effects with the video stream. This can provide a fully automated pet video editing method for pet owners, without the need for pet owners to operate, and at the same time make the edited video more in line with the pet's state, that is, the effect (special effect) presented in the video echoes the pet's current emotion, so as to provide the attractiveness and fun of the automatically edited video.

[0039] Secondly, the special effects implemented in the present application are added directly to the original video (ie, the target video), which has better visual effects than the existing special effects that can only be added to the video transitions.

[0040] Optionally, the above step inputs the patch of each frame image into Transformer for feature extraction to obtain the embedding value corresponding to the patch of each frame image, including: inputting the patch of each frame image into Transformer for feature extraction to obtain the initial embedding value corresponding to the patch of each frame image; based on the relative position relationship between the patches of each frame image, re-embedding the initial embedding value to obtain the embedding value corresponding to the patch of each frame image.

[0041] It should be noted that, considering that the position of the pet in the picture is closely related to the relative position relationship between the patches divided by the original image, the embodiment of the present invention uses relative position coding to perform secondary embedding on the initial embedding value to more accurately capture the position information of the pet on the image.

[0042] Optionally, the number of patches in each frame is ;in, Indicates the number of height directions, represents the number in the width direction; based on the relative position relationship between the patches of each frame image, the initial embedding value is re-embedded to obtain the embedding value corresponding to the patch of each frame image, including: based on the initial embedding value corresponding to the patch of each frame image, the number of rows is , the number of columns is Generate two grids respectively and splice them; the two grids are symmetrical; flatten the spliced ​​two grids to obtain a flattened one-dimensional tensor; flatten the one-dimensional tensor according to the number of rows , the number of columns is Reshape it and add an empty dimension to the second and third dimensions respectively, thereby generating two different three-dimensional tensors; subtract the two different three-dimensional tensors to obtain a relative position table; wherein the relative position table corresponds to the relative position relationship between the patches of the frame image; based on the relative position table, re-embed the initial embedding value to obtain the embedding value corresponding to the patch of each frame image.

[0043] The following is a detailed description. The patch of each frame image is input into Transformer for feature extraction. After obtaining the initial embedding value corresponding to the patch of each frame image, the embedding sequence corresponding to each frame image can be generated. ,like: .

[0044] in, Indicates patches; Indicates the first patch, Indicates Patches, ; express The corresponding initial embedding value, represents a placeholder vector (described in the subsequent embodiment); for F frame images, the total embedded sequence (uppercase) can be represented as: ;in, represents the embedded sequence of the first frame image, represents the embedded sequence of the second frame image, Represents the embedded series of the F-th frame image.

[0045] Then, the initial embedding value corresponding to the patch of each frame image is calculated according to the number of rows. , the number of columns is Generate two grids respectively and splice them; the two grids are symmetrical; flatten the spliced ​​two grids to obtain a flattened one-dimensional tensor; flatten the one-dimensional tensor according to the number of rows , the number of columns is Reshape and add an empty dimension to the second and third dimensions respectively, thereby generating two different three-dimensional tensors, namely and , the three-dimensional tensor and Do subtraction to get the relative position table.

[0046] Then, the initial embedding value is re-embedded based on the relative position table, and the following formula can be used for reference: ; in, express Embedding value after secondary embedding; , , Both represent learnable weights, , , Both indicate Patch and The relative position relationship of the patches (all through the three-dimensional tensor and Subtract and get the relative position table to determine), superscript , , denote the query vector, key vector and value vector respectively, Indicates The initial embedding value of the patch, in this formula Start traversing from 1, represents transpose, Indicates the length of the sequence.

[0047] Optionally, the method provided in the embodiment of the present application also includes: calculating the attention matrix of each frame image based on the embedding value corresponding to the patch of each frame image; counting the standard deviation value of the attention matrix; in response to the standard deviation value being greater than a threshold, determining that a pet exists in the frame image; or in response to the standard deviation value being less than or equal to a threshold, determining that no pet exists in the frame image.

[0048] Specifically, get the embedding value corresponding to the patch output in the previous step ( ), then calculate the attention matrix for each frame image : The expression is: ; Here and All come from itself, Indicates transpose.

[0049] Then the standard deviation of the attention matrix is ​​counted. If the standard deviation is greater than the threshold (such as setting the threshold to 5), it is considered that there is a pet in the video clip corresponding to the embedding sequence of the frame image. The principle is: when a time series is informative, the temporal attention score will be highly activated in several time clips. On the contrary, if a time series is non-informative, the attention score is more likely to be evenly distributed throughout the video clip.

[0050] Also, the previous placeholder can be modified at this time The value of a specific position in the (the modified position in the embodiment is the first position, which is set to 1 if a pet exists and to 0 if not).

[0051] It should be noted that in order to determine whether there is a pet in the captured video frame, the embodiment of the present application adopts a judgment method based on whether the standard deviation of the attention matrix meets the threshold. The advantage of this method is that it does not require an additional detection model, and the intermediate quantity during behavior detection can be used to complete the judgment of whether a pet exists, thereby improving the model utilization rate.

[0052] In other words, by editing the video in a new way to determine whether a pet exists, the utilization rate of the model is improved, and the editing method reduces the storage requirements for the user-side device (the application point of the present invention can be used in combination with the user's shooting device).

[0053] Optionally, adding the target special effect to the target video stream includes: deleting image frames in which no pet exists from the target video stream to obtain an updated video stream; and adding the target special effect to the updated video stream.

[0054] That is, when there is no image frame of the pet, it can be deleted to reduce the storage requirements of the user-side device.

[0055] Optionally, the above steps add the target special effects to the target video stream, including: deleting image frames in which no pets exist from the target video stream; after deleting the image frames in which no pets exist from the target video stream, screening out image frames with consecutive same emotional states, and adding the target special effects to the image frames with consecutive same emotional states.

[0056] For example, if there are 6 consecutive happy frames in 10 frames, the target special effects are added to these 6 frames. In this way, the continuity of adding special effects is ensured, and the visual effect of the video is improved.

[0057] Optionally, the above step inputs the embedding value corresponding to the patch of each frame image into the recognition model, and outputs the identified emotional state of the pet, including: inputting the embedding value corresponding to the patch of each frame image into the recognition model, and outputting the identified emotional state of the pet, and the pet's location information; wherein the pet's location information includes the center point coordinates of the pet, and the pet's detection box.

[0058] Optionally, after obtaining the embedding value corresponding to the patch of each frame image, the method also includes: generating an embedding sequence based on the embedding value corresponding to the patch of each frame image; wherein the first bit of the embedding sequence is a placeholder vector; the first bit of the placeholder vector is used to characterize whether a pet exists; the second bit of the placeholder vector is used to characterize the emotional state of the pet, the third and fourth bits of the placeholder vector are used to characterize the center point coordinates of the pet, and the fifth and sixth bits of the placeholder vector are used to characterize the detection frame of the pet.

[0059] The first bit of the placeholder vector is used to represent whether a pet exists. If a pet exists, it is set to 1, and if not, it is set to 0. The second bit of the placeholder vector is used to represent the emotional state of the pet, such as happy 1, angry 2, and depressed 3. The third and fourth bits of the placeholder vector are used to represent the coordinates of the pet's center point, i.e., the horizontal and vertical coordinates. The fifth and sixth bits of the placeholder vector are used to represent the pet's detection box, i.e., the length and width.

[0060] Since all information is set in the placeholder vector, subsequent devices can directly perform corresponding operations based on the placeholder vector, which improves efficiency. For example, they can obtain the placeholder of each video, discard the video whose first placeholder is 0, and determine the target special effects based on information such as emotional position.

[0061] Optionally, the above step of adding the target special effect to the target video stream includes: determining an adding position of the target special effect based on the position information of the pet, and adding the target special effect to the target video stream according to the adding position.

[0062] In one embodiment, the target special effect can be added above the pet. In one embodiment, the target special effect can be added below the pet, which is not limited in this application.

[0063] In addition, in the embodiments of the present application, the CLIP (Contrastive Language-Image Pre-training) model can be combined to achieve the addition of special effects.

[0064] Specifically, based on the CLIP model, a text template that identifies the emotional state of the pet and the central position of the pet and combines preset values ​​is used as the input of the CLIP model. The text template used in the embodiment is as follows: "The ___ that can represent ___emotions must be centered on ___ and cannot cover images within the ___ range of the center."

[0065] Among them, the first underline is the recognized emotional state of the pet, the second underline is some preset special effects, such as flowers, cakes, etc., the third underline is the coordinates of the pet's center position, and the fourth underline is the size of the pet, that is, the detection frame.

[0066] After obtaining the special effect image through the CLIP model, it is added to the original image to obtain the target video frame, and finally the video frame is exported to obtain the video required by the end user. If multiple text templates are provided for the same video frame, multiple clipped videos with special effects can be produced for users to choose from.

[0067] In addition, embodiments of the present invention may be linked with the user's terminal devices via the Internet, including but not limited to pet-specific photography equipment, mobile phones, computers, etc.

[0068] See also Figure 3Based on the same inventive concept, the present application provides an automatic editing system 300 for pet videos, including: an acquisition module 301, used to acquire a target video stream; wherein the target video stream is a video stream containing a pet collected by a camera device, and the target video stream includes F frame images; F is a positive integer; a position and action recognition module 302, used to split each frame image into multiple patches; input the patch of each frame image into the Transformer for feature extraction, and obtain the embedding value corresponding to the patch of each frame image; wherein the embedding value is used to capture the position information and action information of the pet in the image; an emotion recognition module 303, used to input the embedding value corresponding to the patch of each frame image into a recognition model, and output the recognized emotional state of the pet; wherein the recognition model includes a multi-head attention module and a multi-layer perceptron arranged in sequence; an automatic editing module 304, used to filter out a target special effect that matches the recognized emotional state of the pet from a pre-built special effect library; and add the target special effect to the target video stream.

[0069] See also Figure 4 Based on the same inventive concept, the present application embodiment provides a module frame of an electronic device 400 that applies the above method. The electronic device 400 includes: at least one processor 401 ( Figure 4 Only one is shown in the figure), a memory 402, a computer program 403 stored in the memory 402 and executable on at least one processor 401, and when the processor 401 executes the computer program 403, the steps of the method in any of the foregoing embodiments are implemented.

[0070] The electronic device 400 may be a server, a personal computer, a notebook computer, etc.

[0071] Those skilled in the art will understand that Figure 4 The electronic device 400 is merely an example and does not constitute a limitation on the electronic device 400 , and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0072] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0073] In some embodiments, the memory 402 may be an internal storage unit of the electronic device 400, such as a hard disk or memory of the electronic device 400. In other embodiments, the memory 402 may also be an external storage device of the electronic device 400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 400. Further, the memory 402 may also include both an internal storage unit of the electronic device 400 and an external storage device.

[0074] It should be noted that the above-mentioned systems, devices, etc. are based on the same concept as the method embodiments of the present application. The modules designed for the systems, the steps performed by the devices, and the technical effects brought about can all be found in the method embodiment section and will not be repeated here.

[0075] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0076] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0077] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0079] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0080] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0081] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for automatically editing pet videos, characterized in that: include: Obtain a target video stream; wherein the target video stream is a video stream containing a pet captured by a camera device, and the target video stream includes F frame images; F is a positive integer; Split each frame into multiple patches; Input the patch of each frame image into Transformer for feature extraction, and obtain the embedding value corresponding to the patch of each frame image; wherein the embedding value is used to capture the position information and action information of the pet in the image; Input the embedding value corresponding to the patch of each frame of the image into the recognition model, and output the recognized emotional state of the pet; wherein the recognition model includes a multi-head attention module and a multi-layer perceptron arranged in sequence; Filtering a target special effect that matches the identified emotional state of the pet from a pre-built special effect library; The target special effect is added to the target video stream.

2. The automatic editing method of pet videos according to claim 1, characterized in that: The patch of each frame image is input into the Transformer for feature extraction, and the embedding value corresponding to the patch of each frame image is obtained, including: Input the patch of each frame image into Transformer for feature extraction, and obtain the initial embedding value corresponding to the patch of each frame image; Based on the relative position relationship between the patches of each frame image, the initial embedding value is re-embedded to obtain the embedding value corresponding to the patch of each frame image.

3. The automatic editing method of pet videos according to claim 2, characterized in that: The number of patches in each frame is ;in, Indicates the number of height directions, represents the number in the width direction; the initial embedding value is embedded again based on the relative position relationship between the patches of each frame image to obtain the embedding value corresponding to the patch of each frame image, including: The initial embedding value corresponding to the patch of each frame image is divided into rows. , the number of columns is Generate two grids respectively and splice them; the two grids are in a symmetrical relationship; Flatten the two spliced ​​grids to obtain a flattened one-dimensional tensor; The flattened one-dimensional tensor is divided into , the number of columns is Reshape and add an empty dimension to the second and third dimensions respectively, thus generating two different three-dimensional tensors; Subtract two different three-dimensional tensors to obtain a relative position table; wherein the relative position table corresponds to the relative position relationship between the patches of the frame image; Based on the relative position table, the initial embedding value is embedded twice to obtain the embedding value corresponding to the patch of each frame image.

4. The automatic editing method of pet videos according to claim 3, characterized in that: The method further comprises: Calculate the attention matrix for each frame based on the embedding value corresponding to the patch of each frame; Calculate the standard deviation of the attention matrix; In response to the standard deviation being greater than a threshold, determining that a pet exists in the frame image; or In response to the standard deviation being less than or equal to the threshold, it is determined that no pet exists in the frame image.

5. The automatic editing method of pet videos according to claim 4, characterized in that: Adding the target special effect to the target video stream includes: Deleting image frames where no pet exists from the target video stream to obtain an updated video stream; The target special effect is added to the updated video stream.

6. The automatic editing method of pet videos according to claim 4, characterized in that: Adding the target special effect to the target video stream includes: Deleting image frames where no pet is present from the target video stream; After deleting the image frames in which no pet exists from the target video stream, the image frames in the same consecutive emotional state are screened out, and the target special effect is added to the image frames in the same consecutive emotional state.

7. The automatic editing method of pet videos according to claim 4, characterized in that: The embedding value corresponding to the patch of each frame of the image is input into the recognition model, and the recognized emotional state of the pet is output, including: The embedded value corresponding to the patch of each frame image is input into the recognition model, and the recognized emotional state of the pet and the location information of the pet are output; wherein the location information of the pet includes the coordinates of the center point of the pet and the detection frame of the pet.

8. The automatic editing method of pet videos according to claim 7, characterized in that: After obtaining the embedding value corresponding to the patch of each frame of the image, the method further includes: Generate an embedding sequence based on the embedding values ​​corresponding to the patches of each frame image; Among them, the first bit of the embedded sequence is a placeholder vector; the first bit of the placeholder vector is used to characterize whether a pet exists; the second bit of the placeholder vector is used to characterize the emotional state of the pet, the third and fourth bits of the placeholder vector are used to characterize the center point coordinates of the pet, and the fifth and sixth bits of the placeholder vector are used to characterize the detection box of the pet.

9. The automatic editing method of pet videos according to claim 7, characterized in that: The adding the target special effect to the target video stream comprises: Based on the position information of the pet, the adding position of the target special effect is determined, and according to the adding position, the target special effect is added to the target video stream.

10. An automatic editing system for pet videos, characterized in that: include: An acquisition module, used for acquiring a target video stream; wherein the target video stream is a video stream containing a pet collected by a camera device, and the target video stream includes F frame images; F is a positive integer; A position and action recognition module is used to split each frame of the image into multiple patches; the patches of each frame of the image are input into the Transformer for feature extraction to obtain the embedding value corresponding to the patch of each frame of the image; wherein the embedding value is used to capture the position information and action information of the pet in the image; An emotion recognition module, used to input the embedding value corresponding to the patch of each frame of the image into the recognition model, and output the recognized emotional state of the pet; wherein the recognition model includes a multi-head attention module and a multi-layer perceptron arranged in sequence; The automatic editing module is used to select target special effects that match the recognized emotional state of the pet from a pre-built special effects library; and add the target special effects to the target video stream.

Images