Video processing method, device and electronic equipment

Abstract

The present specification discloses a video processing method, device and electronic device, the method comprising: obtaining a target video to be processed; performing a first segmentation process and a second segmentation process on a target object respectively; wherein a target picture is selected from the target video; according to the result of the first segmentation process and the result of the second segmentation process, a key picture is selected from the target picture; based on a preset spatiotemporal memory network, according to the key picture and the designated segmentation result of the key picture, a final segmentation process is performed on the target object in the target video; wherein the designated segmentation result is the result of the first segmentation process or the result of the second segmentation process.

Description

Technical Field

[0001] The present specification relates to the field of video processing technology, and more specifically, to a video processing method, a live video processing method, a monitoring video processing method, a video processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Video human segmentation is to segment the human body area from each frame of the video, where the human body area includes the human body area, handheld objects and human appendages area.

[0003] Human body segmentation is a common task in computer vision and plays an important role in computers' understanding of human activities. It is widely used in video processing, video surveillance, human pose estimation, 3D modeling, and personal entertainment.

[0004] Existing unsupervised video object segmentation methods are only suitable for simple video scenes, such as scenes with large color differences between people and backgrounds, simple clothing, no handheld objects or accessories, etc. If the scene is slightly more complicated, the segmented area will flicker and the segmentation effect will be poor. Summary of the invention

[0005] One purpose of this specification is to provide a new technical solution for segmenting target objects in a video.

[0006] According to a first aspect of the present specification, a video processing method is provided, comprising:

[0007] Get the target video to be processed;

[0008] Performing a first segmentation process and a second segmentation process on a target object in a target picture respectively; wherein the target picture is selected from the target video;

[0009] Selecting a key picture from the target picture according to a result of the first segmentation process and a result of the second segmentation process;

[0010] Based on a preset spatiotemporal memory network, a final segmentation process is performed on the target object in the target video according to the key image and the designated segmentation result of the key image; wherein the designated segmentation result is a result of the first segmentation process or a result of the second segmentation process.

[0011] Optionally, performing a first segmentation process on the target object in the target image includes:

[0012] Identifying an area in the target image that contains a target object as a target object area;

[0013] A first segmentation process is performed on the target object in the target object area corresponding to the target image to obtain a result of the first segmentation process.

[0014] Optionally, the identifying a region in the target image that contains a target object as the target object region includes:

[0015] Identify a region in the target image that contains an object of a target category as a target category region; wherein the target category is a category to which the target object belongs;

[0016] A target category area that meets the preset conditions is selected as the target object area.

[0017] Optionally, performing a second segmentation process on the target object in the target image includes:

[0018] Based on a preset spatiotemporal memory network, according to a designated picture and a result of the first segmentation process of the designated picture, a second segmentation process is performed on a target object in a second picture to obtain a result of the second segmentation process of the second picture; wherein the designated picture is any of the target pictures, and the second picture is a picture in the target picture other than the designated picture;

[0019] The selecting a key picture from the target picture according to the result of the first segmentation process and the result of the second segmentation process comprises:

[0020] The key picture is selected from the second picture according to the result of the first segmentation process and the result of the second segmentation process.

[0021] Optionally, the performing a second segmentation process on the target object in the second picture based on the preset spatiotemporal memory network according to the specified picture and the result of the first segmentation process of the specified picture, and obtaining the result of the second segmentation process of the second picture includes:

[0022] Using a second picture adjacent to the designated picture as the next picture;

[0023] Based on the spatiotemporal memory network, according to the designated picture and the result of the first segmentation process of the designated picture, a second segmentation process is performed on the target object in the next picture to obtain a result of the second segmentation process of the next picture;

[0024] Taking the next picture as a processed picture;

[0025] Using a second picture adjacent to the processed picture and not subjected to the second segmentation process as the current picture;

[0026] Based on the spatiotemporal memory network, according to the result of the first segmentation process between the designated image and the designated image, and the result of the second segmentation process between the processed image and the processed image, a second segmentation process is performed on the target object in the current image to obtain a result of the second segmentation process of the current image;

[0027] The current picture is used as the processed picture.

[0028] Optionally, selecting a key picture from the target picture according to a result of the first segmentation process and a result of the second segmentation process includes:

[0029] Determine the similarity between the result of the first segmentation process and the result of the second segmentation process of the target image;

[0030] The key picture is selected from the target picture according to the similarity.

[0031] Optionally, selecting the key picture from the target picture according to the similarity includes:

[0032] Selecting a target picture whose similarity is greater than or equal to a preset similarity threshold as the key picture; or,

[0033] Sort the target images according to the similarity, and obtain the sort value of each frame of the target image;

[0034] The target picture whose ranking value is within a set range is selected as the key picture.

[0035] Optionally, the performing final segmentation processing on the target object in the target video based on the preset spatiotemporal memory network according to the key picture and the specified segmentation result of the key picture includes:

[0036] Based on the spatiotemporal memory network, according to the key picture and the specified segmentation result of the key picture, a final segmentation process is performed on the target object in the first picture to obtain a final segmentation process result of the first picture; wherein the first picture is a picture in the target video other than the key picture;

[0037] The designated segmentation result of the key picture is used as the result of the final segmentation processing of the key picture.

[0038] Optionally, also include:

[0039] Get the preset background video;

[0040] A new video is obtained according to the result of the final segmentation processing of the background video and the target video.

[0041] Optionally, also include:

[0042] Obtaining bullet screen information of the target video;

[0043] In the process of displaying the target video, the corresponding barrage information is displayed under the result of the final segmentation processing of the target video.

[0044] According to a second aspect of the present specification, a method for processing a live video is provided, comprising:

[0045] Get live video;

[0046] Performing a first segmentation process and a second segmentation process on a target object in a target picture respectively; wherein the target picture is selected from the live video;

[0047] Selecting a key picture from the target picture according to a result of the first segmentation process and a result of the second segmentation process;

[0048] Based on a preset spatiotemporal memory network, a final segmentation process is performed on the target object in the live video according to the key image and the designated segmentation result of the key image; wherein the designated segmentation result is a result of the first segmentation process or a result of the second segmentation process.

[0049] According to a third aspect of this specification, a method for processing a surveillance video is provided, comprising:

[0050] Get surveillance video;

[0051] Performing a first segmentation process and a second segmentation process on the target object in the target image respectively; wherein the target image is selected from the surveillance video;

[0052] Selecting a key picture from the target picture according to a result of the first segmentation process and a result of the second segmentation process;

[0053] Based on a preset spatiotemporal memory network, the target object in the surveillance video is finally segmented according to the key image and the designated segmentation result of the key image; wherein the designated segmentation result is the result of the first segmentation process or the result of the second segmentation process.

[0054] According to a fourth aspect of the present specification, a video processing method is provided, comprising:

[0055] Providing a first entry for uploading a target video;

[0056] Acquire the target video to be processed through the first entrance;

[0057] Performing a first segmentation process and a second segmentation process on a target object in a target picture respectively; wherein the target picture is selected from the target video;

[0058] Selecting a key picture from the target picture according to a result of the first segmentation process and a result of the second segmentation process;

[0059] providing a second entry for selecting a result of the first segmentation process or a result of the second segmentation process;

[0060] Obtaining the selected segmentation result through the second entry as a designated segmentation result;

[0061] Based on a preset spatiotemporal memory network, a final segmentation process is performed on the target object in the target video according to the key picture and the designated segmentation result of the key picture.

[0062] According to a fifth aspect of the present specification, a video processing device is provided, including:

[0063] A video acquisition module is used to acquire the target video to be processed;

[0064] An initial segmentation module, used to perform a first segmentation process and a second segmentation process on a target object in a target picture, wherein the target picture is selected from the target video;

[0065] A picture selection module, used for selecting a key picture from the target picture based on the result of the first segmentation process and the result of the second segmentation process;

[0066] The final segmentation module is used to perform final segmentation processing on the target object in the target video based on the preset spatiotemporal memory network and according to the key picture and the specified segmentation result of the key picture; wherein the specified segmentation result is the result of the first segmentation processing or the result of the second segmentation processing.

[0067] According to a sixth aspect of this specification, there is provided an electronic device, including:

[0068] The device according to the fifth aspect of this specification; or,

[0069] A processor and a memory, wherein the memory is used to store instructions, and the instructions are used to control the processor to execute the method according to any one of the first to fourth aspects of this specification.

[0070] According to a seventh aspect of the present specification, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any one of the first to fourth aspects of the present specification is implemented.

[0071] In an embodiment of the present specification, a key picture is selected by segmenting the target picture twice, and the target video is segmented according to the key picture and the specified segmentation result of the key picture to obtain the final segmentation result of the target video. In this way, the influence of the inaccurate result of the segmentation process used as a reference on the result of the final segmentation process can be reduced, so that the final segmentation result of each frame of the target video has a smooth transition, avoiding the problem of segmentation flickering, improving the robustness of the final segmentation result of the video, achieving a smooth and high-precision video segmentation effect, and improving the user experience. Moreover, in the storage unit of the spatiotemporal memory network, only the key picture and the specified segmentation result of the key picture can be stored, without storing the target video segmented pictures and the corresponding final segmentation result. In this way, the consumption of the storage unit can be reduced, and the time for the final segmentation process of the target object in the target video can be reduced.

[0072] Other features and advantages of the present specification will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the specification and, together with the description, serve to explain the principles of the specification.

[0074] Figure 1a is a block diagram of an example of a hardware configuration of an electronic device that can be used to implement an embodiment of the present invention.

[0075] Figure 1b is a block diagram of another example of a hardware configuration of an electronic device that can be used to implement an embodiment of the present invention;

[0076] Figure 2a A schematic diagram of a scene showing a method for processing a video according to an embodiment of the present specification.

[0077] Figure 2b A schematic diagram showing a video processing method according to an embodiment of the present specification.

[0078] Figure 3 A schematic flow chart of a video processing method according to an embodiment of the present specification is shown.

[0079] Figure 4 A schematic diagram showing a display interface of an electronic device according to an embodiment of the present specification is shown.

[0080] Figure 5a A schematic diagram showing an existing display interface for barrage information is shown.

[0081] Figure 5bA schematic diagram of a display interface for barrage information of an embodiment of this specification is shown.

[0082] Figure 6 A schematic flow chart of a method for processing live video according to an embodiment of the present specification is shown.

[0083] Figure 7 A schematic flow chart of a monitoring video processing method according to an embodiment of the present specification is shown.

[0084] Figure 8 A flowchart of a method for processing live video according to another embodiment of the present specification is shown.

[0085] Fig. 9 A block diagram showing a video processing apparatus according to an embodiment of the present specification.

[0086] Fig.10 A block diagram showing an example of an electronic device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0087] Various exemplary embodiments of the present specification will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present specification.

[0088] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present specification, its application, or uses.

[0089] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0090] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0091] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0092] <Hardware Configuration>

[0093] Figure 1a and Figure 1b It is a block diagram of the hardware configuration of an electronic device 1000 that can be used to implement the video processing method of any embodiment of the present specification.

[0094] In one embodiment, Figure 1aAs shown, the electronic device 1000 may be a server 1100 .

[0095] Server 1100 provides a computer for processing, database, and communication facilities. Server 1100 can be a monolithic server or a distributed server across multiple computers or computer data centers. The server can be of various types, such as but not limited to, a network server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an interactive server, a database server, or a proxy server. In some embodiments, each server may include hardware, software, or an embedded logic component or a combination of two or more such components for executing the appropriate functions supported or implemented by the server. For example, a server such as a blade server, a cloud server, etc., or a server group consisting of multiple servers, which may include one or more of the above-mentioned types of servers, etc.

[0096] In this embodiment, the server 1100 can be as follows Figure 1a As shown, it includes a processor 1110 , a memory 1120 , an interface device 1130 , a communication device 1140 , a display device 1150 , and an input device 1160 .

[0097] In this embodiment, the server 1100 may also include a speaker, a microphone, etc., which are not limited here.

[0098] The processor 1110 may be a dedicated server processor, or a desktop processor or a mobile processor that meets the performance requirements, which is not limited here. The memory 1120 includes, for example, ROM (read-only memory), RAM (random access memory), non-volatile memory such as a hard disk, etc. The interface device 1130 includes, for example, various bus interfaces, such as a serial bus interface (including a USB interface), a parallel bus interface, etc. The communication device 1140 is capable of wired or wireless communication, for example. The display device 1150 is, for example, a liquid crystal display, an LED display, a touch display, etc. The input device 1160 may include, for example, a touch screen, a keyboard, etc.

[0099] In this embodiment, the memory 1120 of the server 1100 is used to store instructions, which are used to control the processor 1110 to operate to at least perform the method according to any embodiment of this specification. A technician can design instructions according to the scheme disclosed in this specification. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0100] Despite Figure 1a , multiple devices of the server 1100 are shown; however, this specification may only involve some of the devices, for example, the server 1100 only involves the memory 1120 and the processor 1110.

[0101] In one embodiment, the electronic device 1000 may be a terminal device 1200 such as a PC or a laptop computer used by an operator, which is not limited here.

[0102] In this embodiment, refer to Figure 1b As shown, the terminal device 1200 may include a processor 1210, a memory 1220, an interface device 1230, a communication device 1240, a display device 1250, an input device 1260, a speaker 1270, a microphone 1280, and the like.

[0103] The processor 1210 may be a mobile version processor. The memory 1220 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 1230 may include, for example, a USB interface, a headphone interface, etc. The communication device 1240 may, for example, be capable of wired or wireless communication. The communication device 1240 may include a short-range communication device, for example, any device for short-range wireless communication based on short-range wireless communication protocols such as Hi link protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 1240 may also include a remote communication device, for example, any device for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 1250 may, for example, be a liquid crystal display, a touch display, etc. The input device 1260 may include, for example, a touch screen, a keyboard, etc. The user may input / output voice information through the speaker 1270 and the microphone 1280.

[0104] In this embodiment, the memory 1220 of the terminal device 1200 is used to store instructions, which are used to control the processor 1210 to operate to at least perform the method according to any embodiment of this specification. A technician can design instructions according to the scheme disclosed in this specification. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0105] Despite Figure 1b Multiple devices of the terminal device 1200 are shown in the figure, but this specification may only involve some of the devices, for example, the terminal device 1200 only involves the memory 1220, the processor 1210 and the display device 1250.

[0106] <Scenario Example>

[0107] Figure 2a The diagram is a scene diagram of a method for processing a video according to an embodiment of the present specification.

[0108] like Figure 2a As shown, the shooting terminal can obtain the target video to be processed by shooting; the electronic device obtains the target video from the shooting terminal, and performs final segmentation processing on the target object in the target video to obtain the final segmentation processing result of the target video.

[0109] Specifically, Figure 2b As shown, the electronic device can be selected N frames of target pictures from the target video, perform a first segmentation process on the target object in the target picture, and obtain the result of the first segmentation process of the target picture; perform a second segmentation process on the target object in the target picture, and obtain the result of the second segmentation process of the target picture; select a key picture from the target picture according to the result of the first segmentation process and the result of the second segmentation process; based on a preset spatiotemporal memory network, perform a final segmentation process on the target object in the target video according to the key picture and the result of the first segmentation process or the result of the second segmentation process of the key picture, and obtain the result of the final segmentation process of the target video.

[0110] Through the embodiments of the present specification, the key picture is selected based on the results of the two segmentation processes of the target picture, and the target video is segmented according to the key picture and the specified segmentation results of the key picture to obtain the final segmentation processing results of the target video. In this way, the influence of the inaccurate results of the segmentation processing used as a reference on the results of the final segmentation processing can be reduced, so that the final segmentation processing results of each frame of the target video are smoothly transitioned, avoiding the problem of segmentation flickering, improving the robustness of the final segmentation processing results of the video, achieving a smooth and high-precision video segmentation effect, and improving the user experience. Moreover, in the storage unit of the spatiotemporal memory network, only the key picture and the specified segmentation processing results of the key picture can be stored, without the need to store the target video's segmented pictures and the corresponding final segmentation processing results. In this way, the consumption of the storage unit can be reduced, and the time for the final segmentation processing of the target object in the target video can be reduced.

[0111] <Method Example>

[0112] <First Embodiment>

[0113] In this embodiment, a video processing method is provided. The method may be implemented by an electronic device. The electronic device may be Figure 1a The server 1100 shown, or Figure 1b Terminal device 1200 is shown.

[0114] according to Figure 3 As shown, the video processing method of this embodiment may include the following steps S1000 to S4000:

[0115] Step S1000, obtaining a target video to be processed.

[0116] In an embodiment of the present specification, the target video may be a video to be segmented, which may include at least one frame of picture.

[0117] In one embodiment of the present specification, before executing step S1000, the target video may be pre-stored in the electronic device executing the present embodiment, or may be stored in other electronic devices, where the other electronic devices may be electronic devices other than the electronic device executing the present embodiment.

[0118] When the target video is pre-stored in the electronic device executing this embodiment, the target video to be processed may be directly obtained by obtaining the locally stored target video. When the target video is pre-stored in other electronic devices, the electronic device executing this embodiment may obtain the target video from the other electronic devices.

[0119] In one embodiment of the present specification, the target video may be any one of a film and television video, a game video, a surveillance video, a live video, an advertising video, and an online education video.

[0120] Step S2000 , performing a first segmentation process and a second segmentation process on a target object in a target picture selected from a target video.

[0121] In one embodiment of the present specification, the target object may be pre-set according to the application scenario or specific requirements. For example, the target object may be a human body, a cat, an elephant, or a camel.

[0122] The target image may be multiple frames of images in the target video. For example, the target video may be multiple consecutive frames of images captured from the target video, or multiple frames of images randomly sampled from the target video, or all images in the target video.

[0123] In this embodiment, the manner of performing the first segmentation process on the target object in each frame of the target picture to obtain the result of the first segmentation process is the same, and therefore, only one frame of the target picture is used as an example for description below.

[0124] In this embodiment, when performing the first segmentation processing on the target object in the target image, it can be based on a preset segmentation algorithm, and the first segmentation processing can be performed only on the target image itself, and the result of the first segmentation processing will not affect the results of the first segmentation processing of other target images.

[0125] The preset segmentation algorithm may be pre-set according to an application scenario or specific requirements. For example, the segmentation algorithm may be hrnet or deeplab v3+.

[0126] In one embodiment of the present specification, performing a first segmentation process on a target object in a target image may include steps S2100 to S2200 as follows:

[0127] Step S2100: identifying an area in the target image that contains a target object as a target object area.

[0128] In one embodiment of the present specification, the target object region may be a minimum rectangular region including the target object. For different target images, since the sizes of the included target objects may be different, the sizes of the obtained target object regions may also be different.

[0129] Specifically, the target object area may be selected by a rectangular frame for identifying the target object. The size of the rectangular frame may be adaptively adjusted according to the size of the target object in the target image.

[0130] In one embodiment of the present specification, identifying a region in a target image that contains a target object as a target object region may include steps S2110 to S2120 as follows:

[0131] Step S2110 , identifying an area in the target image that contains an object of the target category as a target category area.

[0132] The target category is the category to which the target object belongs. For example, the target object may be the human body of a specified user, the target category may be human body, and correspondingly, the objects of the target category may include the human bodies of all users. For another example, the target object may be a white cat, the target category may be cat, and correspondingly, the objects of the target category may include cats of all colors.

[0133] The method of identifying a region containing an object of the target category may refer to the aforementioned method of identifying a region containing a target object, which will not be described in detail herein.

[0134] By performing object recognition of the target category on the target image, one target category region, multiple target category regions, or zero target category region may be obtained.

[0135] In this embodiment, each target category region includes a complete object belonging to the target category, that is, each target category region uniquely corresponds to a complete object belonging to the target category.

[0136] For example, if the target image is subjected to target category object recognition, and the target image includes object A and object B belonging to the target category, then a target category region A corresponding to object A and a target category region B corresponding to object B can be obtained. In the target category region A, a complete object A and at most a partial object B may be included; in the target category region B, a complete object B and at most a partial object A may be included.

[0137] Step S2120: Select a target category area that meets the preset conditions as the target object area.

[0138] In one embodiment of the present specification, the target object may be pre-set according to an application scenario or specific requirements. Then, selecting a target category region that meets the preset conditions as the target object region may specifically include: selecting a target category region corresponding to the target object as the target object region.

[0139] In another embodiment of the present specification, the target object may also be obtained according to the target video.

[0140] In one example, selecting a target category area that meets preset conditions as a target object area may specifically include: for all objects belonging to the target category contained in the target image, determining a first number of images containing the corresponding target category area; selecting objects whose first number exceeds a preset first number threshold as target objects; and using the target category area corresponding to the target object as the target object area. The first number threshold may be set in advance according to an application scenario or specific needs. For example, when the number of target images is 100, the first number threshold may be 90.

[0141] For example, when the target video is a film and television video, the target category may be a person, and the target object may be a protagonist. The region containing the person in the target image may be selected as the target category region; and then the first number of target category regions corresponding to each person is determined. In general, the protagonist appears more frequently in the video, so the characters whose first number exceeds the first number threshold may be selected as the protagonist, and the target object region corresponding to the protagonist may be used as the target object region.

[0142] In another example, selecting a target category area that meets preset conditions as a target object area may specifically include: for all objects belonging to the target category contained in the target image, determining a second number of images whose positions of the target category area in the target image are within a specified range, selecting objects whose second number exceeds a preset second number threshold as target objects; and using the target category area corresponding to the target object as the target object area. The second number threshold may be set in advance according to an application scenario or specific needs. For example, when the number of target images is 100, the second number threshold may be 40.

[0143] In this embodiment, the position of the target category area in the corresponding image may be: the position of the center of the target category area in the corresponding image. The specified range may be pre-set according to the application scenario or specific requirements, for example, the specified range may be a circular area with the center of the image as the center and the specified radius as the radius, wherein the specified radius may be pre-set according to the application scenario or specific requirements, for example, may be 100 pixels.

[0144] For example, when the target video is a film and television video, the target category may be a person, and the target object may be a protagonist. The region containing the person in the target picture may be selected as the target category region; and then the second number of pictures in which the position of each person in the target picture is within a specified range is determined. Under normal circumstances, the probability frequency of the protagonist appearing within the specified range in the video is high. Therefore, the second number of characters exceeding the second number threshold may be selected as the protagonist, and the target object region corresponding to the protagonist may be used as the target object region.

[0145] In another example, selecting a target category area that meets preset conditions as a target object area may specifically include: for all objects belonging to the target category contained in the target image, determining a third number of images containing the corresponding target category area whose size exceeds a preset size threshold, selecting objects whose third number exceeds a preset third number threshold as target objects; and using the target category area corresponding to the target object as the target object area. The size threshold may be set in advance according to an application scenario or specific needs, for example, the size threshold may be 100 pixels × 100 pixels. The third number threshold may be set in advance according to an application scenario or specific needs. For example, the third number threshold may be 50.

[0146] For example, when the target video is a film and television video, the target category may be a person, and the target object may be a protagonist. The region containing the person in the target picture may be selected as the target category region; and then the third number of pictures whose size of the target category region corresponding to each person exceeds a preset size threshold is determined. In general, the protagonist appears in a video in a larger size, so the third number of characters exceeding the third number threshold may be selected as the protagonist, and the target object region corresponding to the protagonist may be used as the target object region.

[0147] Step S2200: performing a first segmentation process on the target object in the target object area corresponding to the target image to obtain a result of the first segmentation process of the target image.

[0148] In this embodiment, based on a preset segmentation algorithm, the target objects in the target object area corresponding to the target image are respectively subjected to a first segmentation process to obtain a result of the first segmentation process of the target image. The segmentation algorithm may be pre-set according to an application scenario or specific requirements, for example, the segmentation algorithm may be hrnet or deeplab v3+.

[0149] The method of the second division process in this specification is different from the method of the first division process.

[0150] In one embodiment of the present specification, the second segmentation processing method can be based on a preset segmentation algorithm, and the first segmentation processing is performed on the target object in the target object area corresponding to the target image to obtain the result of the second segmentation processing of the target image. The segmentation algorithm can be pre-set according to the application scenario or specific needs. For example, the segmentation algorithm can be deeplab v3+ or hrnet.

[0151] In another embodiment of the present specification, performing a second segmentation process on the target object in the target image may include:

[0152] Based on the preset spatiotemporal memory network, according to the designated picture and the result of the first segmentation processing of the designated picture, a second segmentation processing is performed on the target object in the second picture to obtain the result of the second segmentation processing of the second picture.

[0153] The designated picture is any target picture; and the second picture is a picture in the target picture other than the designated picture.

[0154] In this embodiment, the preset space-time memory network may be a pre-trained space-time memory network (Space-Time Memory network, STM for short) that can be used to segment target objects in a video.

[0155] In one embodiment of the present specification, the designated picture may be pre-set according to an application scenario or specific requirements. For example, the designated picture may be the first frame, the last frame, or an intermediate frame of the target video.

[0156] In one embodiment of the present specification, based on a preset spatiotemporal memory network, a second segmentation process is performed on a target object in a second image according to a specified image and a result of a first segmentation process of the specified image, and the result of the second segmentation process of the second image may include: Figure 4 Steps S2300 to S2800 shown:

[0157] Step S2300: Use the second picture adjacent to the designated picture as the next picture.

[0158] In one embodiment of the present specification, the designated picture and the next picture are adjacent in the target video.

[0159] Taking the target video including 100 frames of pictures as an example, in an embodiment where the designated picture is the first frame of picture, the second frame of picture adjacent to the first frame of picture may be used as the next picture; in an embodiment where the designated picture is the last frame of picture (i.e., the 100th frame of picture), the 99th frame of picture adjacent to the 100th frame of picture may be used as the next picture; in an embodiment where the designated picture is the 20th frame of picture, the 19th frame of picture and the 21st frame of picture adjacent to the 20th frame of picture may be used as the next picture.

[0160] In the case where the next picture includes two frames of pictures, the following steps S2400 to S2800 may be performed for each frame of the next picture respectively until all the second pictures are segmented.

[0161] Step S2400, based on the spatiotemporal memory network, according to the specified picture and the result of the first segmentation processing of the specified picture, a second segmentation processing is performed on the target object in the next picture to obtain the result of the second segmentation processing of the next picture.

[0162] Step S2500, taking the next picture as the processed picture.

[0163] Step S2600: Use the second picture adjacent to the processed picture and not subjected to the second segmentation process as the current picture.

[0164] Step S2700, based on the spatiotemporal memory network, according to the result of the first segmentation processing between the specified picture and the specified picture, and the result of the second segmentation processing between the processed picture and the processed picture, a second segmentation processing is performed on the target object in the current picture to obtain the result of the second segmentation processing of the current picture.

[0165] Step S2800, taking the current picture as the processed picture.

[0166] Steps S2600 to S2800 are repeatedly executed until the target video no longer contains a second picture adjacent to the processed picture and not subjected to the second segmentation process.

[0167] For each current picture, it can be obtained based on a spatiotemporal memory network, according to a specified picture and the result of a first segmentation process of the specified picture, at least one frame of a processed picture that has obtained a result of a second segmentation process, and the corresponding result of a second segmentation process.

[0168] The result of the second segmentation processing of the second picture obtained by this embodiment depends on the result of the first segmentation processing of the designated picture and the result of the second segmentation processing of the second picture that is subjected to the second segmentation processing before the corresponding picture.

[0169] Since the designated picture is pre-set according to the application scenario or specific needs, the segmentation effect of the first segmentation processing result of the designated picture may be poor. Therefore, the following steps S3000 and S4000 can be continued to select the key picture, and perform final segmentation processing on the target object in the target video based on the key picture and the designated segmentation result of the key picture.

[0170] Step S3000: Select a key picture from the target picture according to the result of the first segmentation process and the result of the second segmentation process.

[0171] In one embodiment of the present specification, according to the result of the first segmentation process and the result of the second segmentation process, selecting a key picture from the target picture may include the following steps S3100 to S3200:

[0172] Step S3100, determining the similarity between the result of the first segmentation process and the result of the second segmentation process of the target image.

[0173] Step S3200: Select a key picture from the target picture according to the similarity.

[0174] In one embodiment of the present specification, the method of selecting a key picture according to similarity may include: selecting a target picture whose similarity between a result of the first segmentation process and a result of the second segmentation process is greater than or equal to a preset similarity threshold as the key picture.

[0175] The similarity threshold may be pre-set according to application scenarios or specific requirements. For example, the similarity threshold may be 90%.

[0176] In another embodiment of the present specification, the method of selecting key pictures according to similarity may include: sorting target pictures according to similarity and obtaining the sorting value of each frame of target picture; selecting target pictures whose sorting values ​​are within a set range as key pictures.

[0177] In this embodiment, the target images may be sorted in ascending order or in descending order.

[0178] The setting range may be set in advance according to the sorting method, application scenario or specific requirements. For example, when the sorting method is descending, the setting range may be 1 to 10, and then the 10 target images with the greatest similarity may be selected as key images.

[0179] In an embodiment where a second segmentation process is performed on a target object in a target picture to obtain a result of the second segmentation process of the target picture, the method is as follows: based on a spatiotemporal memory network, according to a target designated picture and a result of the first segmentation process of the designated picture, a second segmentation process is performed on the target object in the second picture to obtain a result of the second segmentation process of the second picture. In the embodiment where a key picture is selected from the target picture according to a result of the first segmentation process and a result of the second segmentation process, the key picture can be selected from the second picture according to a result of the first segmentation process and a result of the second segmentation process of the second picture.

[0180] Specifically, for each frame of the second picture, the similarity between the result of the first segmentation process and the result of the second segmentation process is determined; and a key picture is selected from the second picture according to the similarity.

[0181] Furthermore, a second picture whose similarity is greater than or equal to a preset similarity threshold can be selected as the key picture; or, the second pictures can be sorted according to the similarity, and the sorting value of each frame of the second picture can be obtained, and the second picture whose sorting value is within the set range can be selected as the key picture.

[0182] Step S4000, based on the spatiotemporal memory network, the target object in the target video is finally segmented according to the key picture and the specified segmentation result of the key picture.

[0183] The designated segmentation result may be a result of the first segmentation process or a result of the second segmentation process.

[0184] In this embodiment, the final segmentation processing results of the target video are obtained based on the key picture and the designated segmentation results of the key picture, and are irrelevant to the final segmentation processing results of the already segmented pictures in the target video.

[0185] The spatiotemporal memory network used in step S4000 may be a pre-trained spatiotemporal memory network that can be used to segment the target object in the video. It may be the same as or different from the spatiotemporal memory network used in the second segmentation process, and is not limited here.

[0186] In one embodiment of the present specification, based on the spatiotemporal memory network, according to the key picture and the specified segmentation result of the key picture, performing final segmentation processing on the target object in the target video may include the following steps S4100 to S4200:

[0187] Step S4100, based on the spatiotemporal memory network, according to the key picture and the designated segmentation result of the key picture, the target object in the first picture is finally segmented to obtain the final segmentation result of the first picture. The first picture is a picture in the target video other than the key picture.

[0188] Step S4200: taking the designated segmentation result of the key picture as the result of the final segmentation processing of the key picture.

[0189] Through this embodiment, the final segmentation result of the target video can be obtained.

[0190] Through the embodiments of the present specification, the key picture is selected based on the results of the two segmentation processes of the target picture, and the target video is segmented according to the key picture and the specified segmentation results of the key picture to obtain the final segmentation processing results of the target video. In this way, the influence of the inaccurate results of the segmentation processing used as a reference on the results of the final segmentation processing can be reduced, so that the final segmentation processing results of each frame of the target video are smoothly transitioned, avoiding the problem of segmentation flickering, improving the robustness of the final segmentation processing results of the video, achieving a smooth and high-precision video segmentation effect, and improving the user experience. Moreover, in the storage unit of the spatiotemporal memory network, only the key picture and the specified segmentation processing results of the key picture can be stored, without the need to store the target video's segmented pictures and the corresponding final segmentation processing results. In this way, the consumption of the storage unit can be reduced, and the time for the final segmentation processing of the target object in the target video can be reduced.

[0191] In one embodiment of the present specification, the method may further include:

[0192] A preset background video is obtained; and a new video is obtained according to the result of the final segmentation processing of the background video and the target video.

[0193] The background video may be selected in advance according to an application scenario or specific requirements, and may include at least one frame of background image.

[0194] Specifically, the result of the final segmentation processing corresponding to each frame of the target video may be added to the upper layer of the background image of the corresponding frame to obtain a new video, wherein the arrangement order of the background images corresponding to the result of the final segmentation processing in the new video is the same as the arrangement order of the images corresponding to the result of the final segmentation processing in the target video.

[0195] When the number of background pictures contained in the background video is equal to the number of pictures corresponding to the final segmentation processing results in the target video, the final segmentation processing results corresponding to each frame of the target video can be added to the upper layer of the background picture of the corresponding frame to obtain a new video.

[0196] When the number of background images contained in the background video is less than the number of images corresponding to the final segmentation processing result in the target video, any frame of the background image can be selected for copying, and the copied background image can be inserted into the adjacent position of the corresponding background image until the number of background images is not less than the number of images corresponding to the final segmentation processing result in the target video. Then, the final segmentation processing result corresponding to each frame of the target video is added to the upper layer of the background image of the corresponding frame to obtain a new video.

[0197] In the case where the number of background images contained in the background video is greater than the number of images corresponding to the final segmentation processing results in the target video, continuous background images whose number is equal to the number of images corresponding to the final segmentation processing results in the target video can be selected as target background images according to the application scenario or specific needs. The final segmentation processing result corresponding to each frame of the target video is added to the upper layer of the target background image of the corresponding frame to obtain a new video. For example, the target background image can be selected from the Nth frame of the background video backwards, or from the Nth frame of the background video forwards, where N is a positive integer less than or equal to the total number of background images.

[0198] On the basis of obtaining the new video, the new video can be played in response to a request to play the new video for the user to watch.

[0199] In one example, when a new video is obtained, a request to play the new video may be automatically triggered. Alternatively, a user may trigger a request to play the new video by performing a specified operation through an electronic device.

[0200] Through this embodiment, the final segmentation processing result is added to the upper layer of the background video, and the video background can be changed according to user needs, so that each frame of the new video has a smooth transition, the visual effect is more realistic, and the user experience is better.

[0201] In one embodiment of the present specification, the relative position of each frame of the target video obtained from the final segmentation processing result in the corresponding picture may be fixed.

[0202] Based on this embodiment, the user can also adjust the position of the final segmentation result in the background video.

[0203] Specifically, since the relative positions between the final segmentation processing results are fixed, the position of the final segmentation processing result in any background image in the background video can be adjusted, and the positions of the final segmentation processing results in other background images will be adaptively adjusted accordingly.

[0204] In one embodiment of the present specification, the method may further include:

[0205] Obtain the barrage information of the target video; in the process of displaying the target video, display the corresponding barrage information under the result of the final segmentation processing of the target video.

[0206] Specifically, the final segmentation processing result of each frame in the target video can be displayed on the top layer, the barrage information can be displayed on the lower layer of the final segmentation processing result, and the area of ​​each frame in the target video except the final segmentation processing result can be displayed on the lower layer of the barrage information.

[0207] In this embodiment, by displaying the bullet screen information under the final segmentation processing result of each frame of the target video, it is possible to avoid the bullet screen information blocking the target object in the target video and avoid affecting the user's viewing experience.

[0208] For example, when the content of the currently displayed bullet screen information includes "bullet screen information 1, bullet screen information 2 and bullet screen information 3", and when the target object (human body) in the target video is not segmented, the way to display the bullet screen information in the target video can be as follows: Figure 5a As shown, the bullet comment information is displayed on the upper layer of each frame of the target video, which will block the target object.

[0209] Through the embodiments of this specification, the target object in the target video is segmented, and the final segmentation result can be as follows: Figure 5a and Figure 5b In the case where the bullet screen information is displayed under the final segmentation processing result of each frame of the target video, the display effect of the bullet screen information can be as follows: Figure 5b shown.

[0210] In such Figure 5bIn the example shown, the "info" in "bullet screen information 1" can be displayed on the lower layer of the human body; the "bullet screen" in "bullet screen information 2" can be displayed on the lower layer of the human body; and the "bullet" in "bullet screen information 3" can be displayed on the lower layer of the human body. As the bullet screen information moves, the "bullet screen" in "bullet screen information 1" can be moved out of the display interface; the "bullet screen information 2" is completely exposed; and the "info 3" in "bullet screen information 3" can be displayed on the lower layer of the human body.

[0211] In this way, when the bullet comment information moves to the position of the target object, it is displayed at the lower layer of the result of the final segmentation processing, which can avoid the bullet comment information from blocking the target object in the target video.

[0212] <Second Embodiment>

[0213] In this embodiment, a method for processing live video is provided. The method may be implemented by an electronic device. The electronic device may be Figure 1a The server 1100 shown, or Figure 1b Terminal device 1200 is shown.

[0214] according to Figure 6 As shown, the live video processing method of this embodiment may include the following steps S6100 to S6400:

[0215] Step S6100, obtaining live video.

[0216] The live video in this embodiment may be a video acquired during the live broadcast process, which may include at least one frame of picture.

[0217] Step S6200: Perform a first segmentation process and a second segmentation process on the target object in the target picture selected from the live video.

[0218] The manner of performing the first segmentation processing and the second segmentation processing on the target object in the target image can refer to step S2000 in the aforementioned embodiment, which will not be described in detail here.

[0219] Step S6300: Select a key picture from the target picture according to the result of the first segmentation process and the result of the second segmentation process.

[0220] According to the result of the first segmentation process and the result of the second segmentation process, the method of selecting the key picture from the target picture can refer to step S3000 in the aforementioned embodiment, which will not be described in detail here.

[0221] Step S6400, based on the preset spatiotemporal memory network, the target object in the live video is finally segmented according to the key picture and the specified segmentation result of the key picture.

[0222] The designated segmentation result is a result of the first segmentation process or a result of the second segmentation process.

[0223] Based on the preset spatiotemporal memory network, according to the key picture and the specified segmentation result of the key picture, the method of performing final segmentation processing on the target object in the live video can refer to step S4000 in the aforementioned embodiment, which will not be repeated here.

[0224] <Third Embodiment>

[0225] In this embodiment, a method for processing surveillance video is provided. The method may be implemented by an electronic device. The electronic device may be Figure 1a The server 1100 shown, or Figure 1b Terminal device 1200 is shown.

[0226] according to Figure 7 As shown, the monitoring video processing method of this embodiment may include the following steps S7100 to S7400:

[0227] Step S7100, obtaining surveillance video.

[0228] The monitoring video in this embodiment may be a video collected by a monitoring device installed indoors, outdoors or on a road, and may include at least one frame of picture.

[0229] Step S7200: Perform a first segmentation process and a second segmentation process on the target object in the target picture selected from the surveillance video.

[0230] The manner of performing the first segmentation processing and the second segmentation processing on the target object in the target image can refer to step S2000 in the aforementioned embodiment, which will not be described in detail here.

[0231] Step S7300: Select a key picture from the target picture according to the result of the first segmentation process and the result of the second segmentation process.

[0232] According to the result of the first segmentation process and the result of the second segmentation process, the method of selecting the key picture from the target picture can refer to step S3000 in the aforementioned embodiment, which will not be described in detail here.

[0233] Step S7400, based on the preset spatiotemporal memory network, the target object in the surveillance video is finally segmented according to the key image and the specified segmentation result of the key image.

[0234] The designated segmentation result is a result of the first segmentation process or a result of the second segmentation process.

[0235] Based on the preset spatiotemporal memory network, according to the key picture and the specified segmentation result of the key picture, the method of performing final segmentation processing on the target object in the surveillance video can refer to step S4000 in the aforementioned embodiment, which will not be repeated here.

[0236] <Fourth Embodiment>

[0237] In this embodiment, a video processing method is provided. The method may be implemented by an electronic device. The electronic device may be Figure 1a The server 1100 shown, or Figure 1b Terminal device 1200 is shown.

[0238] according to Figure 8 As shown, the video processing method of this embodiment may include the following steps S8100 to S8700:

[0239] Step S8100, providing a first entry for uploading a target video.

[0240] In this embodiment, the user can upload the target video for segmentation processing through the first entrance.

[0241] Step S8200, obtaining the target video to be processed through the first entry.

[0242] For example, in response to the user clicking the first entry, a video file selection window may be provided, the window including videos and selection buttons that can be provided by the electronic device. The user may select a target video in the window according to actual needs and click the selection button. In response to the user clicking the selection button, the video selected by the user is obtained through the first entry as the target video.

[0243] For another example, the user may directly drag the target video for segmentation processing to the first entrance. In response to the operation, the electronic device obtains the target video to be processed through the first entrance.

[0244] Step S8300: Perform a first segmentation process and a second segmentation process on the target object in the target picture selected from the target video.

[0245] The manner of performing the first segmentation processing and the second segmentation processing on the target object in the target image can refer to step S2000 in the aforementioned embodiment, which will not be described in detail here.

[0246] Step S8400: Select a key picture from the target picture according to the result of the first segmentation process and the result of the second segmentation process.

[0247] According to the result of the first segmentation process and the result of the second segmentation process, the method of selecting the key picture from the target picture can refer to step S3000 in the aforementioned embodiment, which will not be described in detail here.

[0248] Step S8500, providing a second entry for selecting a result of the first segmentation process or a result of the second segmentation process.

[0249] Through the second entrance, the user can select the result of the first segmentation processing or the result of the second segmentation processing according to the application scenario or his own needs.

[0250] For example, in response to a user clicking on a reference to the second entry, a drop-down list containing the results of the first segmentation processing and the results of the second segmentation processing is provided, and the user selects the result of the first segmentation processing or the result of the second segmentation processing by clicking on the result of the first segmentation processing or the result of the second segmentation processing in the drop-down list.

[0251] For another example, the second entry in this embodiment includes: a second entry corresponding to the result of the first segmentation process, and a second entry corresponding to the result of the second segmentation process. The user completes the operation of selecting the result of the first segmentation process or the result of the second segmentation process by clicking the corresponding second entry.

[0252] Step S8600, obtaining the result of the selected segmentation processing through the second entry as the designated segmentation result.

[0253] In this embodiment, the result of the segmentation process selected by the user through the second entrance is used as the designated segmentation result, that is, the designated segmentation result is the result of the first segmentation process or the result of the second segmentation process.

[0254] Step S8700, based on the preset spatiotemporal memory network, the target object in the target video is finally segmented according to the key picture and the specified segmentation result of the key picture.

[0255] Based on the preset spatiotemporal memory network, according to the key picture and the specified segmentation result of the key picture, the method of performing final segmentation processing on the target object in the target video can refer to step S4000 in the aforementioned embodiment, which will not be repeated here.

[0256] <Device Example>

[0257] In this embodiment, a video processing device 6000 is provided, such as Fig. 9As shown, it includes a video acquisition module 6100, an initial segmentation module 6200, a picture selection module 6300 and a final segmentation module 6400. The video acquisition module 6100 is used to acquire a target video to be processed; the initial segmentation module 6200 is used to perform a first segmentation process and a second segmentation process on a target object in a target picture respectively; wherein the target picture is selected from a target video; the picture selection module 6300 is used to select a key picture from the target picture based on the result of the first segmentation process and the result of the second segmentation process; the final segmentation module 6400 is used to perform a final segmentation process on the target object in the target video based on a preset spatiotemporal memory network and according to the key picture and the designated segmentation result of the key picture; wherein the designated segmentation result is the result of the first segmentation process or the result of the second segmentation process.

[0258] In one embodiment of the present specification, performing a first segmentation process on a target object in a target image includes:

[0259] Identify the area in the target image that contains the target object as the target object area;

[0260] A first segmentation process is performed on the target object in the target object area corresponding to the target image to obtain a result of the first segmentation process.

[0261] In one embodiment of the present specification, the target object region in the target image is identified, and the target object region includes:

[0262] Identify a region in the target image that contains an object of a target category as a target category region; wherein the target category is a category to which the target object belongs;

[0263] The target category area that meets the preset conditions is selected as the target object area.

[0264] In one embodiment of the present specification, performing a second segmentation process on a target object in a target image includes:

[0265] Based on a preset spatiotemporal memory network, according to a designated picture and a result of the first segmentation process of the designated picture, a second segmentation process is performed on a target object in a second picture to obtain a result of the second segmentation process of the second picture; wherein the designated picture is any target picture, and the second picture is a picture in the target picture other than the designated picture;

[0266] Selecting a key picture from the target picture according to a result of the first segmentation process and a result of the second segmentation process includes:

[0267] A key picture is selected from the second picture according to a result of the first segmentation process and a result of the second segmentation process.

[0268] In one embodiment of the present specification, based on a preset spatiotemporal memory network, according to a specified image and a result of a first segmentation process of the specified image, a second segmentation process is performed on a target object in a second image, and the result of the second segmentation process of the second image is obtained, including:

[0269] The second picture adjacent to the specified picture is used as the next picture;

[0270] Based on the spatiotemporal memory network, according to the specified picture and the result of the first segmentation processing of the specified picture, a second segmentation processing is performed on the target object in the next picture to obtain the result of the second segmentation processing of the next picture;

[0271] Treat the next image as the processed image;

[0272] Using a second picture adjacent to the processed picture and not subjected to the second segmentation process as the current picture;

[0273] Based on the spatiotemporal memory network, according to the specified image and the result of the first segmentation processing of the specified image, and the processed image and the result of the second segmentation processing of the processed image, a second segmentation processing is performed on the target object in the current image to obtain the result of the second segmentation processing of the current image;

[0274] Use the current image as the processed image.

[0275] In one embodiment of the present specification, the picture selection module 6300 may also be used for:

[0276] Determine the similarity between the result of the first segmentation process and the result of the second segmentation process of the target image;

[0277] Select key images from the target image based on similarity.

[0278] In one embodiment of the present specification, selecting a key picture from a target picture according to similarity includes:

[0279] Selecting a target image whose similarity is greater than or equal to a preset similarity threshold as a key image; or,

[0280] Sort the target images according to their similarity and obtain the ranking value of each frame of the target image;

[0281] Select target images whose ranking values ​​are within the set range as key images.

[0282] In one embodiment of the present specification, the final segmentation module 6400 may also be used to:

[0283] Based on the spatiotemporal memory network, according to the key picture and the specified segmentation result of the key picture, a final segmentation process is performed on the target object in the first picture to obtain a final segmentation process result of the first picture; wherein the first picture is a picture in the target video other than the key picture;

[0284] The specified segmentation result of the key image is used as the result of the final segmentation processing of the key image.

[0285] In one embodiment of the present specification, the video processing device 6000 may further include:

[0286] A module for obtaining a preset background video;

[0287] A module for obtaining a new video based on the results of the final segmentation processing of the background video and the target video.

[0288] In one embodiment of the present specification, the video processing device 6000 may further include:

[0289] A module for obtaining the bullet screen information of a target video;

[0290] A module used to display the corresponding bullet screen information under the final segmentation processing result of the target video during the display of the target video.

[0291] Those skilled in the art will appreciate that the video processing device 6000 can be implemented in various ways. For example, the video processing device 6000 can be implemented by configuring the processor with instructions. For example, the instructions can be stored in a ROM, and when the device is started, the instructions are read from the ROM into a programmable device to implement the video processing device 6000. For example, the video processing device 6000 can be solidified into a dedicated device (such as an ASIC). The video processing device 6000 can be divided into independent units, or they can be combined together for implementation. The video processing device 6000 can be implemented by one of the various implementations described above, or can be implemented by a combination of two or more of the various implementations described above.

[0292] In this embodiment, the video processing device 6000 can have multiple implementation forms. For example, the video processing device 6000 can be a functional module running in any software product or application that provides video processing services, or it can be an external embedded component, plug-in, patch, etc. of these software products or applications, or it can also be these software products or applications themselves.

[0293] <Electronic Equipment Embodiment>

[0294] In this embodiment, an electronic device 7000 is also provided. The electronic device 7000 may be Figure 1a The server 1100 shown may also be Figure 1b The terminal device 1200 is shown.

[0295] On the one hand, the electronic device 7000 may include the aforementioned video processing device 6000, for implementing the video processing method of any embodiment of the present specification.

[0296] On the other hand, Figure 7 As shown, the electronic device 7000 may also include a processor 7100 and a memory 7200, wherein the memory 7200 is used to store executable instructions; the processor 7100 is used to control the electronic device 7000 to execute a method according to any embodiment of this specification under the control of the instructions.

[0297] In this embodiment, the electronic device 7000 may be a smart speaker, earphone, mobile phone, tablet computer, PDA, desktop computer, laptop computer, etc. For example, the electronic device 7000 may be an electronic product with a video processing function.

[0298] <Computer Readable Storage Medium Embodiment>

[0299] In this embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method of any embodiment of this specification is implemented.

[0300] The present specification may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present specification.

[0301] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or a first freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or a first transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0302] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0303] The computer program instructions for performing the operation of this specification can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions, thereby realizing various aspects of this specification.

[0304] Various aspects of this specification are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of this specification. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0305] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, the programmable data processing device and / or the first device to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0306] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0307] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of this specification. In this regard, each box in the flowchart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0308] The embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of this specification is defined by the appended claims.

Claims

1. A video processing method, comprising: Get the target video to be processed; Performing a first segmentation process and a second segmentation process on a target object in a target picture respectively; wherein the target picture is selected from the target video; and the first segmentation process and the second segmentation process are different; Selecting a key picture from the target picture according to a result of the first segmentation process and a result of the second segmentation process; Based on a preset spatiotemporal memory network, a final segmentation process is performed on the target object in the target video according to the key image and the designated segmentation result of the key image; wherein the designated segmentation result is a result of the first segmentation process or a result of the second segmentation process.

2. According to the method of claim 1, performing a first segmentation process on the target object in the target image comprises: Identifying an area in the target image that contains a target object as a target object area; A first segmentation process is performed on the target object in the target object area corresponding to the target image to obtain a result of the first segmentation process.

3. The method according to claim 2, wherein the identifying the area in the target image containing the target object as the target object area comprises: Identify a region in the target image that contains an object of a target category as a target category region; wherein the target category is a category to which the target object belongs; A target category area that meets the preset conditions is selected as the target object area.

4. According to the method of claim 1, performing a second segmentation process on the target object in the target image comprises: Based on a preset spatiotemporal memory network, according to a designated picture and a result of the first segmentation process of the designated picture, a second segmentation process is performed on a target object in a second picture to obtain a result of the second segmentation process of the second picture; wherein the designated picture is any of the target pictures, and the second picture is a picture in the target picture other than the designated picture; The selecting a key picture from the target picture according to the result of the first segmentation process and the result of the second segmentation process comprises: The key picture is selected from the second picture according to the result of the first segmentation process and the result of the second segmentation process.

5. According to the method of claim 4, the step of performing a second segmentation process on the target object in the second image based on the preset spatiotemporal memory network according to the specified image and the result of the first segmentation process of the specified image, and obtaining the result of the second segmentation process of the second image comprises: Using a second picture adjacent to the designated picture as the next picture; Based on the spatiotemporal memory network, according to the designated picture and the result of the first segmentation process of the designated picture, a second segmentation process is performed on the target object in the next picture to obtain a result of the second segmentation process of the next picture; Taking the next picture as a processed picture; Using a second picture adjacent to the processed picture and not subjected to the second segmentation process as the current picture; Based on the spatiotemporal memory network, according to the result of the first segmentation process between the designated image and the designated image, and the result of the second segmentation process between the processed image and the processed image, a second segmentation process is performed on the target object in the current image to obtain a result of the second segmentation process of the current image; The current picture is used as the processed picture.

6. The method according to claim 1, wherein selecting a key picture from the target picture according to the result of the first segmentation process and the result of the second segmentation process comprises: Determine the similarity between the result of the first segmentation process and the result of the second segmentation process of the target image; The key picture is selected from the target picture according to the similarity.

7. The method according to claim 6, wherein selecting the key picture from the target picture according to the similarity comprises: Selecting a target image whose similarity is greater than or equal to a preset similarity threshold as the key image; Alternatively, the target images are sorted according to the similarity, and a sorting value of each frame of the target image is obtained; The target picture whose ranking value is within a set range is selected as the key picture.

8. According to the method of claim 1, the step of performing final segmentation processing on the target object in the target video based on the preset spatiotemporal memory network and the key picture and the specified segmentation result of the key picture comprises: Based on the spatiotemporal memory network, according to the key picture and the specified segmentation result of the key picture, a final segmentation process is performed on the target object in the first picture to obtain a final segmentation process result of the first picture; wherein the first picture is a picture in the target video other than the key picture; The designated segmentation result of the key picture is used as the result of the final segmentation processing of the key picture.

9. The method according to claim 1, further comprising: Get the preset background video; A new video is obtained according to the result of the final segmentation processing of the background video and the target video.

10. The method according to claim 1, further comprising: Obtaining bullet screen information of the target video; In the process of displaying the target video, the corresponding barrage information is displayed under the result of the final segmentation processing of the target video.

11. A method for processing live video, comprising: Get live video; Performing a first segmentation process and a second segmentation process on the target object in the target image respectively; wherein the target image is selected from the live video; and the first segmentation process and the second segmentation process are different; Selecting a key picture from the target picture according to a result of the first segmentation process and a result of the second segmentation process; Based on a preset spatiotemporal memory network, a final segmentation process is performed on the target object in the live video according to the key image and the designated segmentation result of the key image; wherein the designated segmentation result is a result of the first segmentation process or a result of the second segmentation process.

12. A method for processing surveillance video, comprising: Get surveillance video; Performing a first segmentation process and a second segmentation process on the target object in the target image respectively; wherein the target image is selected from the surveillance video; and the first segmentation process and the second segmentation process are different; Selecting a key picture from the target picture according to a result of the first segmentation process and a result of the second segmentation process; Based on a preset spatiotemporal memory network, a final segmentation process is performed on the target object in the surveillance video according to the key image and the designated segmentation result of the key image; wherein the designated segmentation result is a result of the first segmentation process or a result of the second segmentation process.

13. A video processing method, comprising: Providing a first entry for uploading a target video; Acquire the target video to be processed through the first entrance; Performing a first segmentation process and a second segmentation process on a target object in a target picture respectively; wherein the target picture is selected from the target video; and the first segmentation process and the second segmentation process are different; Selecting a key picture from the target picture according to a result of the first segmentation process and a result of the second segmentation process; providing a second entry for selecting a result of the first segmentation process or a result of the second segmentation process; Obtaining the selected segmentation result through the second entry as a designated segmentation result; Based on a preset spatiotemporal memory network, a final segmentation process is performed on the target object in the target video according to the key picture and the designated segmentation result of the key picture.

14. A video processing device, comprising: A video acquisition module is used to acquire the target video to be processed; An initial segmentation module, used to perform a first segmentation process and a second segmentation process on a target object in a target picture, respectively; wherein the target picture is selected from the target video; and the first segmentation process and the second segmentation process are different; A picture selection module, used for selecting a key picture from the target picture based on the result of the first segmentation process and the result of the second segmentation process; The final segmentation module is used to perform final segmentation processing on the target object in the target video based on the preset spatiotemporal memory network and according to the key picture and the specified segmentation result of the key picture; wherein the specified segmentation result is the result of the first segmentation processing or the result of the second segmentation processing.

15. An electronic device, wherein: include: The device according to claim 14; or, A processor and a memory, wherein the memory is used to store instructions, and the instructions are used to control the processor to execute the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 13 when executed by a processor.

Images