Video processing method, device, storage medium and electronic device

Abstract

The present disclosure relates to a video processing method, which makes full use of the association between video frame images, so as to achieve better video instance segmentation effect and improve video instance segmentation efficiency. The method comprises: obtaining a video to be processed; inputting the video to be processed into an instance segmentation model to obtain an instance segmentation result of each frame image in the video to be processed, and the instance segmentation model is used to determine the instance segmentation result of at least one frame image in the video to be processed according to the instance segmentation results of the historical frame images before the image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular, to a video processing method, device, storage medium and electronic device. Background Art

[0002] Video understanding can analyze video content and capture key information in the video. It is widely used in video technology fields such as security monitoring, human behavior analysis, and sports video commentary. Video instance segmentation, as the basis of video understanding tasks, mainly predicts the label of each pixel in the video frame image and distinguishes different individuals belonging to the same category. For example, if a video includes multiple different characters, video instance segmentation can be used to segment the multiple different characters from the video frame image, providing a basis for further video understanding processes such as video target tracking. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a video processing method, the method comprising:

[0005] Get the video to be processed;

[0006] The video to be processed is input into an instance segmentation model to obtain an instance segmentation result for each frame image in the video to be processed. The instance segmentation model is used to determine the instance segmentation result of the image for at least one frame image in the video to be processed based on the instance segmentation results of historical frame images before the image.

[0007] In a second aspect, the present disclosure provides a video processing device, the device comprising:

[0008] An acquisition module is used to acquire the video to be processed;

[0009] A segmentation module is used to input the video to be processed into an instance segmentation model to obtain an instance segmentation result for each frame image in the video to be processed. The instance segmentation model is used to determine the instance segmentation result of the image for at least one frame image in the video to be processed based on the instance segmentation results of historical frame images before the image.

[0010] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when the program is executed by a processing device.

[0011] In a fourth aspect, the present disclosure provides an electronic device, including:

[0012] a storage device having a computer program stored thereon;

[0013] A processing device is used to execute the computer program in the storage device to implement the steps of the method in the first aspect.

[0014] Through the above technical solution, the instance segmentation model can be used to determine the instance segmentation result of at least one frame image in the video to be processed based on the instance segmentation results of the historical frame images before the image. Compared with the method of performing independent instance segmentation for each frame image in the video, the association between the video frame images can be fully utilized to achieve a better video instance segmentation effect. In addition, since instance segmentation is performed in combination with the instance segmentation results corresponding to the historical video frame images before the video frame image, redundant instance segmentation operations can be avoided, thereby improving the efficiency of video instance segmentation.

[0015] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0017] Figure 1 is a flowchart of a video processing method according to an exemplary embodiment of the present disclosure;

[0018] Figure 2 This is a schematic diagram of pixel-by-pixel segmentation based on a rectangular coordinate system;

[0019] Figure 3 This is a schematic diagram of edge segmentation based on polar coordinates;

[0020] Figure 4 is a schematic diagram of an instance segmentation model in a video processing method according to an exemplary embodiment of the present disclosure;

[0021] Figure 5 is a schematic diagram of performing instance segmentation using a video processing method according to an exemplary embodiment of the present disclosure;

[0022] Figure 6 is a block diagram of a video processing device according to an exemplary embodiment of the present disclosure;

[0023] Figure 7It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0025] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0026] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0027] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should also be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0029] Video understanding can analyze video content and capture key information in the video. It is widely used in video technology fields such as security monitoring, human behavior analysis, and sports video commentary. Video instance segmentation, as the basis of video understanding tasks, mainly predicts the label of each pixel in the video frame image and distinguishes different individuals belonging to the same category. For example, if a video includes multiple different characters, video instance segmentation can be used to segment the multiple different characters from the video frame image, providing a basis for further video understanding processes such as video target tracking.

[0030] The inventors have found that in the video processing process in the related art, video instance segmentation is usually performed on each frame of the video separately, that is, when performing instance segmentation on a certain frame of the video, the instance segmentation result corresponding to the previous frame of the frame is not fully utilized, and it is difficult to achieve the optimal segmentation effect. In addition, for video frames with temporal continuity, the contents of the video frames are usually related to each other or have the same parts. If each frame is segmented independently, there are likely to be many redundant instance segmentation operations, which not only affects the instance segmentation efficiency, but also affects the execution efficiency of subsequent video understanding tasks.

[0031] In view of this, the present disclosure provides a video processing method to perform instance segmentation in combination with instance segmentation results corresponding to historical video frame images before the video frame image, making full use of the relationship between video frame images, so as to achieve better video instance segmentation effect and improve video instance segmentation efficiency.

[0032] Figure 1 is a flowchart of a video processing method according to an exemplary embodiment of the present disclosure. Figure 1 , the video processing method comprises:

[0033] Step 101, obtaining a video to be processed.

[0034] For example, obtaining the video to be processed may be in response to a user's video input operation to obtain the video input by the user, or it may be automatically obtaining the video taken by the image acquisition device from the image acquisition device after receiving a video instance segmentation instruction, etc. The embodiments of the present disclosure are not limited to this.

[0035] Step 102, input the video to be processed into an instance segmentation model to obtain an instance segmentation result for each frame image in the video to be processed. The instance segmentation model is used to determine the instance segmentation result of the image for at least one frame image in the video to be processed based on the instance segmentation results of historical frame images before the image.

[0036] Through the above method, the instance segmentation model can be used to determine the instance segmentation result of at least one frame image in the video to be processed based on the instance segmentation results of the historical frame images before the image. Compared with the method of performing independent instance segmentation for each frame image in the video, the association between the video frame images can be fully utilized to achieve a better video instance segmentation effect. In addition, since instance segmentation is performed in combination with the instance segmentation results corresponding to the historical video frame images before the video frame image, redundant instance segmentation operations can be avoided, thereby improving the efficiency of video instance segmentation.

[0037] In order to enable those skilled in the art to better understand the video processing method provided by the present disclosure, the above steps are described in detail with examples below.

[0038] First, the training process of the instance segmentation model is explained.

[0039] For example, the instance segmentation model can be trained based on the sample image and the sample instance segmentation result corresponding to the sample image. The sample instance segmentation result can be a pixel-by-pixel segmentation result based on a rectangular coordinate system, or can be an edge segmentation result based on polar coordinates, which is not limited in the embodiments of the present disclosure. The pixel-by-pixel segmentation result based on a rectangular coordinate system can refer to Figure 2 In this process, a large number of pixels need to be classified, so the instance segmentation result includes the classification information of each pixel. The edge segmentation result based on polar coordinates can be referred to Figure 3 In this process, a ray can be emitted from the center, and the angle and distance of the ray to the edge point can be predicted. Finally, the closed figure obtained by connecting the end points of each ray in sequence is used as the instance segmentation result. Therefore, the instance segmentation result can include the direction information and distance information of each ray.

[0040] In a possible manner, the training step of the instance segmentation model may include: extracting multiple frames of sample images that are continuous in time from a sample video, wherein each frame of the sample image is annotated with a sample instance segmentation result. Then, for at least one frame of the sample image, based on the sample instance segmentation result of the historical frame sample image before the sample image and the feature map of the sample image, determining the predicted instance segmentation result of the sample image, and calculating the loss function based on the predicted instance segmentation result of the sample image and the sample instance segmentation result of the sample image. Finally, adjusting the parameters of the instance segmentation model according to the calculation result of the loss function.

[0041] For example, the historical frame sample image can be the previous frame sample image of the sample image, or can be the previous two frames sample image of the sample image, and so on. The embodiments of the present disclosure are not limited to this, as long as the time position of the historical frame sample image in the video is before the sample image.

[0042] It should be understood that, for the first frame sample image, since there is no historical frame sample image before the sample image, the predicted instance segmentation result of the first frame sample image can be to first randomly predict multiple sample rays emitted from the center point of the instance to be segmented in the first frame image, and then connect the end points of the multiple sample rays to form a closed figure as the predicted instance segmentation result of the first frame sample image, and calculate the loss function based on the predicted instance segmentation result and the sample instance segmentation result of the first frame sample image, and finally adjust the parameters of the instance segmentation model according to the calculation result of the loss function.

[0043] Afterwards, for each sample image frame except the first sample image frame, the predicted instance segmentation result of the sample image can be determined based on the sample instance segmentation result of the historical frame sample image before the sample image and the feature map of the sample image, so as to calculate the loss function based on the predicted instance segmentation result and the corresponding sample instance segmentation result to realize the training of the instance segmentation model. Among them, the feature map can be an image obtained by quantizing the image feature vector of each pixel in the sample image.

[0044] In a possible manner, the sample instance segmentation result may be an instance segmentation result based on polar coordinates, that is, the sample instance segmentation result includes sample ray information, and then the sample instance segmentation adjustment information of the sample image based on polar coordinates may be determined based on the sample instance segmentation result of the historical frame sample image and the feature map of the sample image. Then, the first sample ray in the sample instance segmentation result based on polar coordinates of the historical frame sample image is adjusted according to the sample instance segmentation adjustment information to obtain a second sample ray. Finally, the predicted instance segmentation result of the sample image based on polar coordinates is determined according to the second sample ray.

[0045] For example, the sample instance segmentation adjustment information may be direction adjustment information and / or angle adjustment information for the first sample ray. In a possible manner, the sample instance segmentation adjustment information may be a sample instance segmentation adjustment ray. Considering that the contents included in each frame of the video are usually interrelated or have the same parts, for example, each frame of the video represents different actions of the same person, the center point of the ray emitted during the instance segmentation process does not change, but the length of the ray changes. Therefore, the starting point of the sample instance segmentation adjustment ray may be the end point of the first sample ray, and the direction may be the same as or opposite to that of the first sample ray. Thus, after adjusting the first sample ray according to the sample instance segmentation adjustment information, the first sample ray may change in the direction of the center point or in a direction close to the center point.

[0046] After adjusting the first sample ray according to the sample instance segmentation adjustment information, that is, obtaining the second sample ray, the predicted instance segmentation result of the sample image based on polar coordinates can be determined according to the second sample ray. For example, a closed figure formed by sequentially connecting the end points of the second sample ray can be used as the predicted instance segmentation result of the sample image based on polar coordinates. Then, the loss function can be calculated according to the predicted instance segmentation result and the sample instance segmentation result pre-labeled in the sample image, so as to adjust the parameters of the instance segmentation model according to the calculation result of the loss function to realize the training of the instance segmentation model.

[0047] For example, a schematic diagram of an instance segmentation model is shown in Figure 4 See Figure 4The instance segmentation model includes a neural network. The structure of the neural network can be set according to actual conditions, and the embodiment of the present disclosure does not limit this. Figure 4 , the training process of the instance segmentation model includes: inputting the feature map of the sample image and the first sample ray corresponding to the historical frame sample image into the neural network, the neural network can learn the features of the input feature map and the first sample ray, and output the sample instance segmentation adjustment ray. Then, the instance segmentation model can superimpose the sample instance segmentation adjustment ray on the basis of the first sample ray to obtain the second sample ray corresponding to the sample image, and determine the predicted instance segmentation result of the sample image according to the second sample ray. Finally, the loss function can be calculated according to the predicted instance segmentation result and the sample instance segmentation result corresponding to the sample image to adjust the parameters of the instance segmentation model. At the same time, the sample image can be used as the historical frame sample image of the next frame sample image, that is, the second sample ray corresponding to the sample image can be used in the instance segmentation process of the next frame sample image. Through this continuous iterative training method, the parameters of the instance segmentation model can be optimized and the segmentation accuracy of the instance segmentation model can be improved.

[0048] Through the above method, the model can be trained according to the sample instance segmentation results of the historical frame images before the sample image. Therefore, the trained instance segmentation model can perform instance segmentation on the image in combination with the instance segmentation results of the historical frame images before the image, so as to achieve better instance segmentation effect and improve instance segmentation efficiency. In addition, the instance segmentation model can be trained by the sample instance segmentation results based on polar coordinates. Accordingly, the trained instance segmentation model can perform instance segmentation based on polar coordinates. Compared with the pixel-by-pixel segmentation method based on the rectangular coordinate system, the instance segmentation efficiency can be further improved and the delay between each frame image during the instance segmentation process can be reduced.

[0049] The following describes the process of performing video instance segmentation on the video to be processed using the instance segmentation model trained in the above manner.

[0050] For example, after obtaining the video to be processed, the video to be processed can be input into the instance segmentation model. It should be understood that each frame image in the video to be processed has a time sequence, so a video image sequence composed of multiple frames of images arranged in time sequence can be obtained according to the video to be processed. Therefore, inputting the video to be processed into the trained instance segmentation model can also be inputting the video image sequence corresponding to the video to be processed into the instance segmentation model.

[0051] For example, the trained instance segmentation model can determine the instance segmentation result of the image for at least one frame of image in the video to be processed according to the instance segmentation results of historical frame images before the image.

[0052] As described above, in a possible way, the instance segmentation model can be trained by the sample image and the sample instance segmentation results of the previous frame of the sample image. Therefore, in the application stage, the historical frame image before the image can be the previous frame of the image. Therefore, in the instance segmentation process of the previous and next frames of images, the instance segmentation result of the previous frame can be used as the initial state when the next frame of the instance segmentation is performed, and the association between the two adjacent frames of images is fully utilized, so as to achieve a better video instance segmentation effect and improve the efficiency of video instance segmentation.

[0053] In a possible manner, considering that there is no historical frame image before the first frame image, the trained instance segmentation model can be used for the first frame image in the video to be processed, emitting multiple rays from the center point of the instance to be segmented in the first frame image, and connecting the end points of the multiple rays to form a closed figure as the instance segmentation result of the first frame image. It should be understood that in the training stage, the parameters of the instance segmentation model are adjusted by the result of calculating the loss function through the predicted instance segmentation result of the first frame sample image and the sample instance segmentation result based on polar coordinates. Therefore, in the application stage, the instance segmentation model can output a more accurate instance segmentation result based on polar coordinates for the first frame image of the video to be processed.

[0054] In a possible manner, the instance segmentation result of the historical frame image may include a first ray emitted from the center point of the instance to be segmented. Accordingly, for each frame image except the first frame in the video to be processed, the instance segmentation adjustment information of the image based on polar coordinates can be determined according to the instance segmentation result of the historical frame image based on polar coordinates and the feature map of the image, and then the first ray is adjusted according to the instance segmentation adjustment information to obtain a second ray emitted from the center point of the instance to be segmented, and finally the instance segmentation result of the image based on polar coordinates is determined according to the second ray.

[0055] For example, the instance segmentation adjustment information may be direction adjustment information and / or angle adjustment information for the first ray. In a possible manner, the instance segmentation adjustment information may include an instance segmentation adjustment ray, the starting point of the instance segmentation adjustment ray is the end point of the first ray, and the direction of the instance segmentation adjustment ray is the same as or opposite to the direction of the first ray. Accordingly, adjusting the first ray according to the instance segmentation adjustment information may be: superimposing the instance segmentation adjustment ray on the basis of the first ray, so that the first ray changes in a direction away from the center point or in a direction close to the center point.

[0056] It should be understood that, in the training phase, the instance segmentation model adjusts the first sample ray according to the sample instance segmentation to obtain the second sample ray, thereby obtaining a predicted instance segmentation result for the calculation of the loss function, and adjusting the parameters of the instance segmentation model according to the calculation result of the loss function. Therefore, in the application phase, after the feature map of the image and the instance segmentation result of the historical frame image are input into the trained instance segmentation model, the instance segmentation model can output the corresponding instance segmentation adjustment ray, which can be understood as the correction amount from the ray of the historical frame image based on polar coordinates to the ray of the current frame image based on polar coordinates. Thus, by superimposing the instance segmentation adjustment ray on the ray of the historical frame image based on polar coordinates (i.e., the first ray), the ray of the current frame image based on polar coordinates can be obtained, thereby obtaining the instance segmentation result of the current frame image.

[0057] For example, the instance segmentation results of the historical frame image based on polar coordinates are as follows: Figure 3 As shown, the instance segmentation model can be used to obtain Figure 5 The instance segmentation adjustment ray shown by the dashed line in the middle is superimposed on Figure 5 The historical frame image shown by the solid line in the middle is based on the ray of polar coordinates, and the second ray of the current frame image based on polar coordinates can be obtained. Thus, according to the closed figure formed by sequentially connecting the end points of the second ray, the instance segmentation result of the current frame image can be obtained ( Figure 5 (illustrated by irregular shapes).

[0058] Through the above method, the instance segmentation results of the previous historical frame images of the image can be combined to perform instance segmentation on the image, achieving a better instance segmentation effect and improving the instance segmentation efficiency. In addition, instance segmentation can be performed based on polar coordinates, which can further improve the instance segmentation efficiency and reduce the delay between each frame image during the instance segmentation process compared to the pixel-by-pixel segmentation method based on the rectangular coordinate system.

[0059] Based on the same inventive concept, the embodiment of the present disclosure also provides a video processing device, which can be part or all of the electronic device through software, hardware or a combination of both. Figure 6 , the video processing device 600 comprises:

[0060] An acquisition module 601 is used to acquire a video to be processed;

[0061] The segmentation module 602 is used to input the video to be processed into an instance segmentation model to obtain an instance segmentation result for each frame image in the video to be processed. The instance segmentation model is used to determine the instance segmentation result of the image for at least one frame image in the video to be processed based on the instance segmentation results of historical frame images before the image.

[0062] Optionally, the instance segmentation result of the historical frame image includes a first ray emitted from a center point of the instance to be segmented, and the segmentation module 602 is used to:

[0063] Determining instance segmentation adjustment information of the image based on polar coordinates according to the instance segmentation result of the historical frame image based on polar coordinates and the feature map of the image;

[0064] The first ray is adjusted according to the instance segmentation adjustment information to obtain a second ray emitted from the center point of the instance to be segmented, and an instance segmentation result of the image based on polar coordinates is determined according to the second ray.

[0065] Optionally, the instance segmentation adjustment information includes an instance segmentation adjustment ray, the starting point of the instance segmentation adjustment ray is the end point of the first ray, the direction of the instance segmentation adjustment ray is the same as or opposite to the direction of the first ray, and the segmentation module 602 is used to:

[0066] The instance segmentation adjustment ray is superimposed on the basis of the first ray, so that the first ray changes in a direction away from the center point or in a direction close to the center point.

[0067] Optionally, the historical frame image is a previous frame image of the image.

[0068] Optionally, the segmentation module 602 is used to send out multiple rays from the center point of the instance to be segmented in the first frame image of the video to be processed through the instance segmentation, and use the closed figure formed by connecting the end points of the multiple rays as the instance segmentation result of the first frame image.

[0069] Optionally, the apparatus further comprises the following module for training the instance segmentation model:

[0070] The first training module is used to extract multiple frames of sample images that are continuous in time from the sample video, wherein each frame of the sample image is annotated with a sample instance segmentation result;

[0071] A second training module is used to determine, for at least one frame of sample image, a predicted instance segmentation result of the sample image according to the sample instance segmentation result of a historical frame of sample image before the sample image and a feature map of the sample image, and calculate a loss function according to the predicted instance segmentation result of the sample image and the sample instance segmentation result of the sample image;

[0072] The third training module is used to adjust the parameters of the instance segmentation model according to the calculation result of the loss function.

[0073] Optionally, the second training module is used to:

[0074] Determining sample instance segmentation adjustment information of the sample image based on polar coordinates according to the sample instance segmentation result of the historical frame sample image based on polar coordinates and the feature map of the sample image;

[0075] Adjusting a first sample ray in a sample instance segmentation result of the historical frame sample image based on polar coordinates according to the sample instance segmentation adjustment information to obtain a second sample ray;

[0076] A predicted instance segmentation result of the sample image based on polar coordinates is determined according to the second sample ray.

[0077] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0078] Based on the same inventive concept, an embodiment of the present disclosure further provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the steps of any of the above-mentioned video processing methods are implemented.

[0079] Based on the same inventive concept, the present disclosure also provides an electronic device, including:

[0080] a storage device having a computer program stored thereon;

[0081] A processing device is used to execute the computer program in the storage device to implement the steps of any of the above-mentioned video processing methods.

[0082] Reference below Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0083] like Figure 7As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0084] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0085] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0086] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0087] In some embodiments, any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) can be used for communication, and can be interconnected with any form or medium of digital data communication (e.g., communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0088] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0089] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a video to be processed; inputs the video to be processed into an instance segmentation model to obtain an instance segmentation result of each frame image in the video to be processed, and the instance segmentation model is used to determine the instance segmentation result of the image for at least one frame image in the video to be processed according to the instance segmentation results of historical frame images before the image.

[0090] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0091] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0092] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a module does not, in some cases, limit the module itself.

[0093] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0094] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] According to one or more embodiments of the present disclosure, Example 1 provides a video processing method, the method comprising:

[0096] Get the video to be processed;

[0097] The video to be processed is input into an instance segmentation model to obtain an instance segmentation result for each frame image in the video to be processed. The instance segmentation model is used to determine the instance segmentation result of the image for at least one frame image in the video to be processed based on the instance segmentation results of historical frame images before the image.

[0098] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the instance segmentation result of the historical frame image includes a first ray emitted from the center point of the instance to be segmented, and the determining the instance segmentation result of the image according to the instance segmentation result of the historical frame image before the image includes:

[0099] Determining instance segmentation adjustment information of the image based on polar coordinates according to the instance segmentation result of the historical frame image based on polar coordinates and the feature map of the image;

[0100] The first ray is adjusted according to the instance segmentation adjustment information to obtain a second ray emitted from the center point of the instance to be segmented, and an instance segmentation result of the image based on polar coordinates is determined according to the second ray.

[0101] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, wherein the instance segmentation adjustment information includes an instance segmentation adjustment ray, the starting point of the instance segmentation adjustment ray is the end point of the first ray, the direction of the instance segmentation adjustment ray is the same as or opposite to the direction of the first ray, and adjusting the first ray according to the instance segmentation adjustment information includes:

[0102] The instance segmentation adjustment ray is superimposed on the basis of the first ray, so that the first ray changes in a direction away from the center point or in a direction close to the center point.

[0103] According to one or more embodiments of the present disclosure, Example 4 provides the method of any one of Examples 1-3, wherein the historical frame image is a previous frame image of the image.

[0104] According to one or more embodiments of the present disclosure, Example 5 provides the method of any one of Examples 1-3, wherein the instance segmentation model is used to emit multiple rays from the center point of the instance to be segmented in the first frame image of the video to be processed, and use a closed figure formed by connecting the end points of the multiple rays as the instance segmentation result of the first frame image.

[0105] According to one or more embodiments of the present disclosure, Example 6 provides the method of any one of Examples 1-3, wherein the training step of the instance segmentation model includes:

[0106] Extracting multiple time-continuous sample image frames from the sample video, wherein each sample image frame is annotated with a sample instance segmentation result;

[0107] For at least one frame of sample image, determine a predicted instance segmentation result of the sample image according to a sample instance segmentation result of a historical frame of sample image before the sample image and a feature map of the sample image, and calculate a loss function according to the predicted instance segmentation result of the sample image and the sample instance segmentation result of the sample image;

[0108] The parameters of the instance segmentation model are adjusted according to the calculation result of the loss function.

[0109] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 6, wherein determining the predicted instance segmentation result of the sample image according to the sample instance segmentation result of the historical frame sample image before the sample image and the feature map of the sample image includes:

[0110] Determining sample instance segmentation adjustment information of the sample image based on polar coordinates according to the sample instance segmentation result of the historical frame sample image based on polar coordinates and the feature map of the sample image;

[0111] Adjusting a first sample ray in a sample instance segmentation result of the historical frame sample image based on polar coordinates according to the sample instance segmentation adjustment information to obtain a second sample ray;

[0112] A predicted instance segmentation result of the sample image based on polar coordinates is determined according to the second sample ray.

[0113] According to one or more embodiments of the present disclosure, Example 8 provides a video processing device, the device comprising:

[0114] An acquisition module is used to acquire the video to be processed;

[0115] A segmentation module is used to input the video to be processed into an instance segmentation model to obtain an instance segmentation result for each frame image in the video to be processed. The instance segmentation model is used to determine the instance segmentation result of the image for at least one frame image in the video to be processed based on the instance segmentation results of historical frame images before the image.

[0116] According to one or more embodiments of the present disclosure, Example 9 provides the apparatus of Example 8, wherein the instance segmentation result of the historical frame image includes a first ray emitted from the center point of the instance to be segmented, and the segmentation module is used to:

[0117] Determining instance segmentation adjustment information of the image based on polar coordinates according to the instance segmentation result of the historical frame image based on polar coordinates and the feature map of the image;

[0118] The first ray is adjusted according to the instance segmentation adjustment information to obtain a second ray emitted from the center point of the instance to be segmented, and an instance segmentation result of the image based on polar coordinates is determined according to the second ray.

[0119] According to one or more embodiments of the present disclosure, Example 10 provides the apparatus of Example 9, wherein the instance segmentation adjustment information includes an instance segmentation adjustment ray, the starting point of the instance segmentation adjustment ray is the end point of the first ray, the direction of the instance segmentation adjustment ray is the same as or opposite to the direction of the first ray, and the segmentation module is used to:

[0120] The instance segmentation adjustment ray is superimposed on the basis of the first ray, so that the first ray changes in a direction away from the center point or in a direction close to the center point.

[0121] According to one or more embodiments of the present disclosure, Example 11 provides the apparatus of any one of Examples 8-10, wherein the historical frame image is a previous frame image of the image.

[0122] According to one or more embodiments of the present disclosure, Example 12 provides an apparatus of any one of Examples 8-10, wherein the segmentation module is used to emit multiple rays from the center point of the instance to be segmented in the first frame image of the video to be processed through the instance segmentation, and use a closed figure formed by connecting the end points of the multiple rays as the instance segmentation result of the first frame image.

[0123] According to one or more embodiments of the present disclosure, Example 13 provides the apparatus of any one of Examples 8-10, wherein the apparatus further includes the following module for training the instance segmentation model:

[0124] The first training module is used to extract multiple frames of sample images that are continuous in time from the sample video, wherein each frame of the sample image is annotated with a sample instance segmentation result;

[0125] A second training module is used to determine, for at least one frame of sample image, a predicted instance segmentation result of the sample image according to the sample instance segmentation result of a historical frame of sample image before the sample image and a feature map of the sample image, and calculate a loss function according to the predicted instance segmentation result of the sample image and the sample instance segmentation result of the sample image;

[0126] The third training module is used to adjust the parameters of the instance segmentation model according to the calculation result of the loss function.

[0127] According to one or more embodiments of the present disclosure, Example 14 provides the apparatus of Example 13, wherein the second training module is used for:

[0128] Determining sample instance segmentation adjustment information of the sample image based on polar coordinates according to the sample instance segmentation result of the historical frame sample image based on polar coordinates and the feature map of the sample image;

[0129] Adjusting a first sample ray in a sample instance segmentation result of the historical frame sample image based on polar coordinates according to the sample instance segmentation adjustment information to obtain a second sample ray;

[0130] A predicted instance segmentation result of the sample image based on polar coordinates is determined according to the second sample ray.

[0131] According to one or more embodiments of the present disclosure, Example 15 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-7 when executed by a processing device.

[0132] According to one or more embodiments of the present disclosure, Example 16 provides an electronic device, including:

[0133] a storage device having a computer program stored thereon;

[0134] A processing device is used to execute the computer program in the storage device to implement the steps of any one of the methods described in Examples 1-7.

[0135] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0136] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0137] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A video processing method, characterized in that: The method comprises: Get the video to be processed; Inputting the video to be processed into an instance segmentation model to obtain an instance segmentation result of each frame image in the video to be processed, wherein the instance segmentation model is used to determine the instance segmentation result of the image for at least one frame image in the video to be processed according to the instance segmentation result of a historical frame image before the image, wherein the time position of the historical frame image in the video to be processed is located before the image; The instance segmentation result of the historical frame image includes a first ray emitted from a center point of the instance to be segmented, and determining the instance segmentation result of the image according to the instance segmentation result of the historical frame image before the image includes: Determining instance segmentation adjustment information of the image based on polar coordinates according to the instance segmentation result of the historical frame image based on polar coordinates and the feature map of the image; adjusting the first ray according to the instance segmentation adjustment information to obtain a second ray emitted from the center point of the instance to be segmented, and determining an instance segmentation result of the image based on polar coordinates according to the second ray; The instance segmentation adjustment information includes an instance segmentation adjustment ray, the starting point of the instance segmentation adjustment ray is the end point of the first ray, the direction of the instance segmentation adjustment ray is the same as or opposite to the direction of the first ray, and adjusting the first ray according to the instance segmentation adjustment information includes: The instance segmentation adjustment ray is superimposed on the basis of the first ray, so that the first ray changes in a direction away from the center point or in a direction close to the center point.

2. The method according to claim 1, characterized in that The historical frame image is the previous frame image of the image.

3. The method according to claim 1, characterized in that The instance segmentation model is used to emit multiple rays from the center point of the instance to be segmented in the first frame image in the video to be processed, and use the closed figure formed by connecting the end points of the multiple rays as the instance segmentation result of the first frame image.

4. The method according to claim 1, characterized in that: The training steps of the instance segmentation model include: Extracting multiple time-continuous sample image frames from the sample video, wherein each sample image frame is annotated with a sample instance segmentation result; For at least one frame of sample image, determine a predicted instance segmentation result of the sample image according to a sample instance segmentation result of a historical frame of sample image before the sample image and a feature map of the sample image, and calculate a loss function according to the predicted instance segmentation result of the sample image and the sample instance segmentation result of the sample image; The parameters of the instance segmentation model are adjusted according to the calculation result of the loss function.

5. The method according to claim 4, characterized in that The step of determining the predicted instance segmentation result of the sample image according to the sample instance segmentation result of the historical frame sample image before the sample image and the feature map of the sample image comprises: Determining sample instance segmentation adjustment information of the sample image based on polar coordinates according to the sample instance segmentation result of the historical frame sample image based on polar coordinates and the feature map of the sample image; Adjusting a first sample ray in a sample instance segmentation result of the historical frame sample image based on polar coordinates according to the sample instance segmentation adjustment information to obtain a second sample ray; A predicted instance segmentation result of the sample image based on polar coordinates is determined according to the second sample ray.

6. A video processing device, characterized in that: The device comprises: An acquisition module is used to acquire the video to be processed; A segmentation module, used for inputting the video to be processed into an instance segmentation model to obtain an instance segmentation result of each frame image in the video to be processed, wherein the instance segmentation model is used for determining the instance segmentation result of the image according to the instance segmentation result of a historical frame image before the image for at least one frame image in the video to be processed, wherein the time position of the historical frame image in the video to be processed is located before the image; The instance segmentation result of the historical frame image includes a first ray emitted from the center point of the instance to be segmented, and the segmentation module is used for: Determining instance segmentation adjustment information of the image based on polar coordinates according to the instance segmentation result of the historical frame image based on polar coordinates and the feature map of the image; adjusting the first ray according to the instance segmentation adjustment information to obtain a second ray emitted from the center point of the instance to be segmented, and determining an instance segmentation result of the image based on polar coordinates according to the second ray; The instance segmentation adjustment information includes an instance segmentation adjustment ray, the starting point of the instance segmentation adjustment ray is the end point of the first ray, the direction of the instance segmentation adjustment ray is the same as or opposite to the direction of the first ray, and the segmentation module is used to: The instance segmentation adjustment ray is superimposed on the basis of the first ray, so that the first ray changes in a direction away from the center point or in a direction close to the center point.

7. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 5 are implemented.

8. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 5.

Images