Image processing method, device, equipment and storage medium

Abstract

The embodiments of the present application provide an image processing method, apparatus, device and storage medium, the method comprising: obtaining an image to be processed, the image to be processed including a filled instance and a non-filled instance; extracting a pixel feature map and a connection feature map of the image to be processed, the pixel feature map is used to characterize whether a target pixel in the filled image belongs to a filled instance, and the connection feature map is used to characterize whether a target pixel and a neighboring pixel of the target pixel belong to the same instance; constructing a connected domain according to the pixel feature map and the connection feature map, and obtaining a target detection frame corresponding to the non-filled instance; intercepting the image to be processed according to the target detection frame, and obtaining a de-filled image corresponding to the image to be processed. The embodiments of the present application determine the target detection frame corresponding to the non-filled instance in the image to be processed based on the characteristics of the pixels, which can improve the accuracy of the recognition of the filled instance, and further ensure that the accurate non-filled instance is intercepted, which is conducive to improving the image effect after removing the filled area.

Description

Technical Field

[0001] The present application relates to image processing technology, and more specifically, to an image processing method, apparatus, device and storage medium. Background Art

[0002] During the delivery process, ad creatives usually need to adapt to different layouts and specifications. For example, if a horizontal ad creative needs to be delivered to a vertical layout, a common practice is to fill the creative with a certain amount of padding. The filled area is called the fill area, and the filled creative is called the fill ad creative.

[0003] Due to the existence of the padding area, padding advertising materials (videos, pictures) will have a certain impact on the advertising effect. On the one hand, the effective area of ​​the advertising material is small. For example, after a 720*1280 vertical video is filled with 1280*720 horizontal padding material, its effective area is only 1 / 3 of the original. A large amount of padding area not only affects the user's perception, but the noise and interference it brings also make it difficult to understand the subsequent advertising material content and extract the advertising material features. On the other hand, if the padding-based size transformation is directly performed on the padding advertising material, the original video material content will be further compressed, resulting in information overload, which will seriously affect the experience. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, device and storage medium, thereby at least to a certain extent solving the problem of images containing filled instances. The target detection frame corresponding to the non-filled instances in the image to be processed is determined based on the characteristics of the pixels, which can improve the accuracy of filling instance recognition and ensure that accurate non-filled instances are captured, which is beneficial to improving the image effect after removing the filled area.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to a first aspect of an embodiment of the present application, there is provided an image processing method, the method comprising:

[0007] Acquire an image to be processed, wherein the image to be processed includes a filled instance and a non-filled instance, wherein the filled instance represents an image of a filled area in the image to be processed, and the non-filled instance represents an image of a non-filled area in the image to be processed;

[0008] Extracting a pixel feature map and a connection feature map of the image to be processed, wherein the pixel feature map is used to characterize whether a target pixel in the filled image belongs to the filled instance, and the connection feature map is used to characterize whether the target pixel and its neighboring pixels belong to the same instance;

[0009] Constructing a connected domain according to the pixel feature map and the connection feature map to obtain a target detection box corresponding to the non-filled instance;

[0010] The image to be processed is intercepted according to the target detection frame to obtain a defilled image corresponding to the image to be processed.

[0011] According to a second aspect of an embodiment of the present application, there is provided an image processing device, the device comprising:

[0012] An image acquisition unit, used for acquiring an image to be processed, wherein the image to be processed includes a filled instance and a non-filled instance, wherein the filled instance represents an image of a filled area in the image to be processed, and the non-filled instance represents an image of a non-filled area in the image to be processed;

[0013] A feature extraction unit, used to extract a pixel feature map and a connection feature map of the image to be processed, wherein the pixel feature map is used to characterize whether a target pixel in the filled image belongs to the filled instance, and the connection feature map is used to characterize whether the target pixel and its neighboring pixels belong to the same instance;

[0014] An object detection frame acquisition unit, configured to construct a connected domain according to the pixel feature map and the connection feature map, and obtain an object detection frame corresponding to the non-filled instance;

[0015] The image capture unit is used to capture the image to be processed according to the target detection frame to obtain a defilled image corresponding to the image to be processed.

[0016] In some embodiments of the present application, based on the above solution, the feature extraction unit is configured as follows:

[0017] Performing multi-level convolution coding on the image to be processed to obtain a multi-level feature map;

[0018] Performing a first convolution operation on the feature maps of the preset multiple levels with the convolution kernel of the first preset channel respectively, and performing feature fusion on the feature maps after the first convolution operation to obtain the pixel feature map;

[0019] A second convolution operation is performed on the feature maps of the preset multiple levels respectively with the convolution kernel of the second preset channel, and feature fusion is performed on the feature maps after the second convolution operation to obtain the connection feature map.

[0020] In some embodiments of the present application, based on the foregoing solution, the preset multiple levels of feature maps include the feature map of the last level among the multiple levels of feature maps, and the device further includes:

[0021] A dilated convolution unit, configured to perform parallel sampling on the feature map of the last level using a first preset number of dilated convolutions to obtain a first preset number of sampled feature maps;

[0022] The atrous convolution unit is further used to cascade the first preset number of sampled feature maps to obtain an updated feature map of the final level.

[0023] In some embodiments of the present application, based on the above solution, the device further includes:

[0024] An image extraction unit, used for extracting multiple frames of images to be processed from the video to be processed;

[0025] The target detection frame acquisition unit is further used to process the multiple frames of images to be processed to obtain target detection frames of the multiple frames of images to be processed respectively;

[0026] The target detection frame acquisition unit is further used to determine the target detection frame of each frame image in the to-be-processed video according to the target detection frames of the multiple frames of images to be processed;

[0027] The image capture unit is further used to capture each frame of the video to be processed according to the target detection frame of each frame of the video to be processed, so as to obtain a de-filled video corresponding to the video to be processed.

[0028] In some embodiments of the present application, based on the above solution, the image extraction unit is configured as follows:

[0029] If the duration of the video to be processed is greater than the first preset duration, intercepting a video of the second preset duration in the video to be processed to obtain an intercepted video;

[0030] A plurality of frames of images are extracted at equal intervals from the captured video to obtain the plurality of frames of images to be processed.

[0031] In some embodiments of the present application, based on the above solution, the target detection frame acquisition unit is configured as follows:

[0032] If the position coordinates of the target detection frames of at least two frames of the images to be processed are the same, determining the target detection frames of the at least two frames of the images to be processed as the confident target detection frames;

[0033] According to the number of the confident target detection frames, the target detection frame of each frame image in the video is determined.

[0034] In some embodiments of the present application, based on the above solution, the target detection frame acquisition unit is further configured as follows:

[0035] If the multiple frames of images to be processed include only one of the confident target detection frames, the confident target detection frame is determined as the target detection frame of each frame image in the video;

[0036] If the multiple frames of images to be processed include at least two of the confident target detection frames, respectively intercepting the videos of the respective shots included in the video to be processed to obtain intercepted videos corresponding to the respective shots;

[0037] Extracting multiple frames of images to be processed from the intercepted videos corresponding to the respective shots, to obtain images to be processed corresponding to the respective shots;

[0038] Processing the to-be-processed images corresponding to the respective lenses to determine target detection frames of the to-be-processed images corresponding to the respective lenses;

[0039] According to the target detection frames of the to-be-processed images corresponding to the respective shots, target detection frames of the respective frame images in the videos of the respective shots are determined.

[0040] According to a third aspect of the embodiments of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method as described in the above embodiments is implemented.

[0041] According to the fourth aspect of the embodiments of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the electronic device, the one or more processors implement the image processing method as described in the above embodiments.

[0042] The embodiment of the present application can improve the accuracy of filled instance recognition by extracting the pixel feature map and connection feature map of the image to be processed, and determining the target detection box corresponding to the non-filled instance in the image to be processed based on the characteristics of the pixels, thereby ensuring that accurate non-filled instances are captured, which is beneficial to improving the image effect after removing the filled area.

[0043] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0045] Figure 1A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0046] Figure 2 A schematic flow chart of an image processing method according to an embodiment of the present application is shown.

[0047] Figure 3 A schematic diagram of an image to be processed according to an embodiment of the present application is shown.

[0048] Figure 4 A schematic diagram showing the relationship between a target pixel and neighboring pixels according to an embodiment of the present application is shown.

[0049] Figure 5 A schematic flow chart of a method for extracting pixel feature maps and connecting feature maps according to an embodiment of the present application is shown.

[0050] Figure 6 A structural schematic diagram of an image processing model according to an embodiment of the present application is shown.

[0051] Figure 7 A schematic diagram of an image processing process according to an embodiment of the present application is shown.

[0052] Figure 8 A schematic flow chart of another image processing method according to an embodiment of the present application is shown.

[0053] Fig. 9 A schematic diagram of another image processing process according to an embodiment of the present application is shown.

[0054] Fig.10 A block diagram of an image processing device according to an embodiment of the present application is shown.

[0055] Fig.11 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0056] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0057] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0058] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0059] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be cascaded or partially cascaded, so the actual execution order may change according to actual conditions.

[0060] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.

[0062] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0063] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0064] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0065] The technical solution of the embodiment of the present application mainly relates to the field of computer vision processing of artificial intelligence technology. For details, please refer to Figure 1 A schematic diagram of an exemplary system architecture that applies the technical solution of an embodiment of the present application is shown.

[0066] like Figure 1 As shown, the system architecture 100 may include a terminal device 110 (eg Figure 1 The embodiment of the present invention is a method for providing a communication link between the terminal device and the server 130. The network 120 is used to provide a medium for a communication link between the terminal device and the server 130. The network 120 may include various connection types, such as a wired communication link, a wireless communication link, and the like.

[0067] In one embodiment of the present application, the server 130 may obtain an image to be processed from a terminal device, and extract a pixel feature map and a connection feature map of the image to be processed, wherein the pixel feature map is used to characterize whether a target pixel in the filled image belongs to a filled instance, and the connection feature map is used to characterize whether the target pixel and its neighboring pixels belong to the same instance; a connected domain is constructed based on the pixel feature map and the connection feature map to obtain a target detection frame corresponding to a non-filled instance; the image to be processed is intercepted based on the target detection frame to obtain a de-filled image corresponding to the image to be processed.

[0068] It should be noted that the image processing method provided in the embodiment of the present application can be executed by the server 130, and accordingly, the image defilling device is generally arranged in the server 130. However, in other embodiments of the present application, the terminal device can also have similar functions as the server, so as to execute the image processing method provided in the embodiment of the present application.

[0069] It should also be noted that Figure 1 The number of terminal devices, networks and servers in the description is only illustrative. According to the implementation requirements, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this, and this application does not limit it here.

[0070] It should be explained that cloud computing as described above is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, and expanded at any time. By establishing a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform, various types of virtual resources are deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0071] The technical solution of the embodiment of the present application is described in detail below:

[0072] Figure 2 FIG. 1 is a flow chart of an image processing method according to an embodiment of the present application. The image processing method may be executed by a computing and processing device, such as a server. Figure 2 As shown, the method includes at least the following steps.

[0073] Step 210: Acquire an image to be processed, wherein the image to be processed includes a filled instance and a non-filled instance. The filled instance represents an image of a filled area in the image to be processed, and the non-filled instance represents an image of a non-filled area in the image to be processed.

[0074] Optionally, the image to be processed can be a horizontal image in the advertising field, an image frame contained in a short video, etc. The following takes a horizontal image in the advertising field as an example to illustrate that if a horizontal image advertising material needs to be delivered to a vertical format, the image usually needs to be filled to a certain extent. The filled part of the image after filling is the filled instance, and the original part is the non-filled instance. In the specific implementation, solid color filling (such as pure black, pure white), Gaussian blur filling and other technical means can be used to fill the image. In order to better utilize the space of the filled instance, subtitles, stickers and other elements can also be added to the filled instance.

[0075] The image after filling processing has the characteristics of obvious filling edge characteristics and obvious horizontal or vertical distribution of filled instances and non-filled instances. Figure 3 A schematic diagram of an image to be processed according to an embodiment of the present application is shown. Figure 3 As shown, the middle part of the image to be processed is a non-filled instance, and the left and right parts are filled instances. The edge between the filled instance and the non-filled instance is obvious, and the filled instance is rectangular and horizontally distributed on the left or right side of the non-filled instance.

[0076] Step 220: extracting a pixel feature map and a connection feature map of the image to be processed, wherein the pixel feature map is used to characterize whether a target pixel in the filled image belongs to a filled instance, and the connection feature map is used to characterize whether a target pixel and its neighboring pixels belong to the same instance.

[0077] In a specific implementation, the pixel feature map can be a two-channel feature map (the total number of channels is 2), one channel represents whether the target pixel is a positive pixel (a pixel belonging to a filled instance), and the other channel represents whether the target pixel is a negative pixel (a pixel belonging to a non-filled instance).

[0078] Figure 4 FIG. 4 shows a schematic diagram of the relationship between a target pixel and neighboring pixels according to an embodiment of the present application. Figure 4 As shown, the embodiment of the present application connects the feature map to represent the target pixel and its neighboring pixels in 8 directions (upper left, upper right, upper left, right, lower left, lower right) (corresponding to Figure 4 1, 2, 3, 4, 5, 6, 7, 8). Accordingly, the connection feature map can be 8 two-channel connection feature maps (the total number of channels is 16), which respectively characterize the connection relationship between the target pixel and its neighbor pixels in 8 directions. One channel in the connection feature map corresponding to each direction characterizes whether the target pixel and the neighbor pixel in that direction are positively connected (the target pixel and the neighbor pixel in that direction belong to the same instance), and the other channel characterizes whether the target pixel and the neighbor pixel in that direction are negatively connected (the target pixel and the neighbor pixel in that direction do not belong to the same instance).

[0079] Step 230: construct a connected domain based on the pixel feature map and the connection feature map to obtain the target detection box corresponding to the non-filled instance.

[0080] In one embodiment of the present application, firstly, the pixel feature map with a channel number of 2 is classified by a first preset threshold (determining whether the target pixel is a positive pixel or a negative pixel) to obtain a positive pixel set, and the connection feature map with a channel number of 16 is classified by a second preset threshold (determining whether the target pixel and its neighboring pixels are positively connected or negatively connected) to obtain a positive connection set; then, based on the rule of constructing a connected domain based on adjacent pixels (when at least one connection between two adjacent pixels in the positive pixel set is a positive connection, then the two adjacent pixels constitute a connected domain), the connected domain of all pixels is constructed in the positive pixel set and the positive connection set by using a union-find method to obtain a segmentation map of the filled instance; finally, the boundingRect() of OpenCV can be used to obtain the minimum right rectangle covering the segmentation map, i.e., the target detection frame. The target detection frame can be expressed as (x, y, w, h), where x and y represent the coordinates of the center point of the target detection frame, respectively, and w and h represent the width and height of the target detection frame, respectively.

[0081] Step 240: intercepting the image to be processed according to the target detection frame to obtain a defilled image corresponding to the image to be processed.

[0082] Optionally, the target detection frame is obtained, that is, the position information of the filling instance is obtained, and the image to be processed can be cropped to obtain a de-filled image that does not contain the filling instance. However, if the position and size of the target detection frame are less than a certain deviation (for example, 5 thousandths) from the boundary of the image to be processed, it means that the target detection frame is close to the boundary of the image to be processed (that is, close to the width and height of the original image), then the boundary of the image to be processed is used as the target detection frame, that is, slight recognition errors of the algorithm are tolerated.

[0083] The embodiment of the present application can improve the accuracy of filled instance recognition by extracting the pixel feature map and connection feature map of the image to be processed, and determining the target detection box corresponding to the non-filled instance in the image to be processed based on the characteristics of the pixels, thereby ensuring that accurate non-filled instances are captured, which is beneficial to improving the image effect after removing the filled area.

[0084] Figure 5 FIG. 1 is a flow chart of a method for extracting pixel feature maps and connecting feature maps according to an embodiment of the present application. Figure 5 As shown, the method includes at least the following steps.

[0085] Step 510: Perform multi-level convolution coding on the image to be processed to obtain a multi-level feature map.

[0086] Optionally, the embodiment of the present application may use the adjusted VGG-16 to perform multi-layer convolution encoding (also referred to as downsampling encoding) on ​​the image to be processed. Figure 6 FIG. 4 shows a schematic diagram of the structure of an image processing model according to an embodiment of the present application. Figure 6 As shown, the backbone network of the model ( Figure 6 The left part of the figure can use the adjusted VGG-16 network structure as the downsampling network. The adjustment of VGG-16 is to adjust the last local area (Block) of the original VGG-16, and set the step size of the pooling layer pool5 in this area to 1; in order to ensure the order between the top, bottom, left and right of the pixels, the two fully connected layers fc6 and fc7 are replaced with convolution operations.

[0087] Step 520: perform a first convolution operation on the feature maps of the preset multiple levels respectively with the convolution kernel of the first preset channel, and perform feature fusion on the feature maps after the first convolution operation to obtain a pixel feature map.

[0088] Specifically, in Figure 6 In the embodiment shown, the right part is an upsampling network. In the embodiment of the present application, the feature maps of adjacent levels in the prediction level are convolved with the 1X1 convolution kernel of the two channels from bottom to top and then fused by unit addition. In order to fuse the feature maps after the fusion of adjacent levels with the upper feature map (the resolution size needs to be the same as the upper feature map), the feature maps after the fusion of adjacent levels need to be upsampled (i.e., the feature maps are enlarged) to sample the feature maps after the fusion of adjacent levels to the resolution size of the upper feature map. In a specific implementation, the bilinear interpolation method can be used to upsample the fused feature map. After the feature maps of the preset levels are fused, they are finally subjected to a 1x1 convolution operation to obtain a two-channel pixel feature map.

[0089] It should be noted that when performing feature fusion, the feature maps of the {conv stage2, conv stage3, conv stage4, conv stage5, fc7} layers can be fused, and the resolution of the pixel feature maps obtained after fusion of these layers is 1 / 2 of the image to be processed; the feature maps of the {conv stage 3, conv stage4, conv stage 5, fc7} layers can also be fused, and the resolution of the pixel feature maps obtained after fusion of these layers is 1 / 4 of the image to be processed. In specific implementations, different layers can be selected for feature fusion according to performance requirements.

[0090] Step 530: perform a second convolution operation on the feature maps of the preset multiple levels respectively with the convolution kernel of the second preset channel, and perform feature fusion on the feature maps after the second convolution operation to obtain a connected feature map.

[0091] It should be noted that the process of obtaining the connection feature map is similar to the process of obtaining the pixel feature map in step 520, but it is necessary to transform the above two-channel 1×1 convolution kernel into a 16-channel convolution kernel.

[0092] The embodiment of the present application uses convolution operation on the image to be processed and identifies the filling instance of the image to be processed based on the pixel unit, which ensures the speed and accuracy and meets the application scenario of image de-filling.

[0093] In some embodiments of the present application, the feature map of the last level in the multi-level feature map (e.g. Figure 6 In the embodiment shown, the fc7 layer may also perform the following processing before performing feature fusion with the feature map of the upper layer:

[0094] Performing parallel sampling on the feature map of the last level using a first preset number of dilated convolutions to obtain a first preset number of sampled feature maps;

[0095] A first preset number of sampled feature maps are cascaded to obtain an updated feature map of the final level.

[0096] Among them, atrous convolutions, also known as dilated convolutions, include a parameter called the "dilation rate", which defines the spacing between values ​​when the convolution kernel processes data. The receptive field of a 3×3 convolution kernel with a dilation rate of 2 is the same as that of a 5×5 convolution kernel, and the 3×3 convolution kernel only requires 9 parameters, that is, under the same computing conditions, the atrous convolution provides a larger receptive field.

[0097] In the embodiment of the present application, multiple dilated convolutions can be used to sample the feature maps of the last level in parallel. In the specific implementation, atrous spatial pyramid pooling (ASPP, which includes 3×3 convolution kernels with expansion rates of 6, 12, and 18) can be used to sample the feature maps of the last level in parallel with dilated convolutions at different sampling rates. The sampled feature maps are cascaded (concatenation is equivalent to splicing the sampled feature maps, increasing the number of channels of the feature maps of the last level). The context of the image to be processed can be captured at multiple scales, and the filled edges between filled instances and non-filled instances can be better detected.

[0098] To intuitively understand the image processing process of this application, Figure 7 FIG. 2 shows a schematic diagram of an image processing process according to an embodiment of the present application. Figure 7As shown in the figure, the image to be processed is encoded by a multi-level convolutional model to obtain a multi-level feature map, and the feature map of the last level is also sampled in parallel by the ASPP module; after the feature maps of the preset levels are fused, the filled pixel recognition (i.e., determining whether the target pixel is a non-filled instance to obtain a positive pixel set) and connection recognition (i.e., determining whether the target pixel and its neighboring pixels belong to the same instance to obtain a positive connection) are performed through two branches respectively, and a heat map corresponding to the filled pixel recognition represents the pixel score map; the eight heat maps corresponding to the connection recognition represent the connection score maps of the pixels in eight directions; and the positive pixel set and the positive connection set obtained by the classification module are then fused (the connected domain is constructed by the two to determine the target detection frame) to obtain the position of the non-filled instance.

[0099] It should be noted that in order for the above image processing model to accurately identify the non-filled area, the image processing model needs to be trained. In a specific implementation, images in advertising materials in the form of images and videos covering multiple major industries can be marked to obtain correctly marked training samples.

[0100] For the pixel characteristics of the target pixel in the image, if the target pixel belongs to the filling instance, its pixel characteristics are marked as positive pixels, and if the target pixel does not belong to the filling instance, its pixel characteristics are marked as negative pixels.

[0101] For the connection characteristics of the target pixel in the image, if the neighboring pixels in a certain direction of the target pixel and its neighborhood belong to a non-filled instance, the connection characteristics of the target pixel in that direction are marked as positive connection, otherwise they are marked as negative connection. In a specific implementation, when determining the connection characteristics of pixels in an image, it is only necessary to determine the characteristics of the pixels at the boundary of the non-filled instance, because the connection between the non-boundary pixel and the 8 pixels in its neighborhood is definitely a positive connection.

[0102] In order to obtain more training samples, data augmentation can also be performed on the original image, and the data augmentation methods include but are not limited to scaling, random rotation, etc. For example, the random rotation method is to randomly rotate the input image with a preset probability, and the rotation angle range value can be [0, π / 2, π, 3π / 2]; then the rotated image is cropped; and finally the cropped image is scaled to a uniform size.

[0103] In the application scenario of advertising, the advertising material may be a padded video, so the video needs to be de-padded. In actual applications, video advertising materials may adopt a single filling method, that is, the filling mode of each frame image in the video is the same (equivalent to the same position of the filling area of ​​each frame image); or in order to enhance the advertising effect at the end of the video, the tail filling method is adopted, that is, the video before the last frame in the video adopts one filling mode, and the last frame adopts another filling mode; in a few cases, the video adopts a variable filling method, that is, different shots in the video adopt different filling modes.

[0104] Figure 8 FIG. 2 is a flow chart of another image processing method according to an embodiment of the present application. Figure 8 As shown, the method includes at least the following steps.

[0105] Step 810: extract multiple frames of images to be processed from the video to be processed.

[0106] As described in the filling method used in the above video, multiple consecutive frames of images in the video use the same filling mode. In order to reduce the number of extracted video frames and thus reduce the amount of image processing calculations, multiple frames of images to be processed can be extracted from the video without affecting the accuracy. The filling mode of other frames in the video is determined based on the filling mode of the extracted images. In the specific implementation, the Opencv index method can be used to decode the video without reading frame by frame, which can effectively reduce the average time consumption.

[0107] Step 820: Process the multiple frames of images to be processed to obtain target detection frames of the multiple frames of images to be processed.

[0108] This step takes steps similar to steps 210-230, which will not be repeated here.

[0109] Step 830: Determine the target detection frame of each frame image in the video to be processed according to the target detection frames of the multiple frames of images to be processed.

[0110] The filling mode of the video is determined according to the target detection frames of the multiple frames of images to be processed. Under different filling modes, the target detection frames of the multiple frames of images to be processed are mapped to the target detection frames of other frames of images in the video in different ways.

[0111] Step 840: intercepting each frame of the video to be processed according to the target detection frame of each frame of the video to be processed, and obtaining a de-filled video corresponding to the video to be processed.

[0112] In order to simply and vividly demonstrate the process of video processing in this application, Fig. 9 FIG. 2 shows another image processing process schematic diagram according to an embodiment of the present application. Fig. 9As shown in the figure, a video with the upper and lower parts filled with advertising content is extracted to obtain 5 frames of images to be processed. After filling and recognition, the 5 frames of images to be processed are used to obtain the target detection frames of the 5 frames of images to be processed. After multi-frame result fusion, the target detection frames of each frame in the processing video are determined according to the target detection frames of the 5 frames of images to be processed, and finally the de-filled video is obtained.

[0113] The embodiment of the present application extracts multiple frames of images to be processed from the video, determines the target detection frames of all frame images in the video, and then obtains the de-filled video corresponding to the video. Without affecting the accuracy, it avoids obtaining the target detection frames of the image frame by frame and reduces the amount of calculation on the number of images.

[0114] In order to reduce the number of images to be processed and thus alleviate the computational pressure of image processing, in some embodiments of the present application, extracting multiple frames of images to be processed in the video to be processed includes:

[0115] If the duration of the video to be processed is greater than the first preset duration, intercepting a video of the second preset duration in the video to be processed to obtain an intercepted video;

[0116] A plurality of frames of images are extracted at equal intervals from the intercepted video to obtain a plurality of frames of images to be processed.

[0117] It should be noted that: videos usually use a single filling method, so in the specific implementation, part of the video can be intercepted from the longer video and multiple frames of images to be processed can be extracted at equal intervals from the intercepted video. For example, for a long video with a duration of more than 30 seconds, only the first 30 seconds of the video are intercepted; then 5 frames are extracted at equal intervals; after determining the filling mode of the 5 frames of images, the filling mode of all frames in the video can be determined. For videos with a duration of 3s-30s, after removing the last two seconds, 5 frames of images to be processed are extracted at equal intervals. For short videos with a duration of less than 3s, only the first frame, the last frame and the middle frame can be extracted, a total of three frames of images to be processed.

[0118] It should be noted that in some cases, the video may also use the tail frame filling method. Therefore, it is also possible to extract the tail frame of each video on the basis of equally spaced frame extraction, obtain the target detection frame of the tail frame separately, and capture the non-filled embodiment of the tail frame.

[0119] further, Figure 8 Step 830 in may include:

[0120] If the position coordinates of the target detection frames of at least two frames of the images to be processed are the same, the target detection frames of the at least two frames of the images to be processed are determined as the confident target detection frames;

[0121] According to the number of confident target detection boxes, the target detection box of each frame image in the video is determined.

[0122] The confident target detection frame is considered to be the target detection frame corresponding to the non-filled instance in some or all images in the video. After obtaining the target detection frames of multiple frames of images to be processed, the number of confident target detection frames can be determined. If the multiple frames of images to be processed extracted from the video only include one confident target detection frame, it means that the video uses a single filling method, and the position coordinates of the target detection of each frame in the video are the same. If it includes at least two confident target detection frames, it means that the video uses a transformation filling method, and the position coordinates of the target detection of different frames in the video may be different. Therefore, the target detection frame of each frame in the video can be further determined according to the number of confident target detection frames.

[0123] Furthermore, according to the number of confident target detection frames, determining the target detection frame of each frame image in the video may include:

[0124] If the multiple frames of images to be processed include a confident target detection frame, the confident target detection frame is determined as the target detection frame of each frame image in the video;

[0125] If the multiple frames of images to be processed include at least two confident target detection frames, the videos of each shot included in the video to be processed are respectively intercepted to obtain the intercepted videos corresponding to each shot;

[0126] Extract multiple frames of images to be processed from the intercepted videos corresponding to each lens, and obtain images to be processed corresponding to each lens;

[0127] Processing the to-be-processed images corresponding to each lens, and determining the target detection frame of the to-be-processed images corresponding to each lens;

[0128] According to the target detection frames of the to-be-processed images corresponding to the respective shots, the target detection frames of the respective frame images in the videos of the respective shots are determined.

[0129] For the case where most of the above videos adopt a single filling method, multiple frames of images to be processed only include one confident target detection box, and the confident target detection box can be directly used as the target detection box of each frame of the image to be processed in the video.

[0130] Corresponding to the situation where a small number of videos use the transformation filling method mentioned above, if multiple frames of images to be processed include at least two confident target detection boxes, it means that the video uses the transformation filling method, and the filling mode of the extracted multiple frames of images to be processed cannot reflect the filling mode of all frame images, and the filling mode of each frame image in the video of each shot in the video is the same. Therefore, the embodiment of the present application splits the video into shots and extracts the images to be processed for each shot of the video.

[0131] It should be noted that since the duration of the video of some shots may be greater than the first preset duration and contain a large number of image frames, when extracting the images to be processed corresponding to the videos of each shot, you can refer to the above-mentioned method of extracting multiple frames of images to be processed in the video to be processed. If the duration of the video of a certain shot is greater than the first preset duration, first capture the video of the shot with a second preset duration to obtain the captured video corresponding to the shot, and then extract multiple frames of images to be processed at equal intervals from the captured video corresponding to the shot to obtain the image to be processed corresponding to the shot.

[0132] In the technical solution of the above-mentioned embodiment of the present application, multiple frames of images to be processed can be extracted, and the target detection frame of the image to be processed can be determined with high precision and efficiency through an image processing method based on pixel units, and then the confidence detection frame can be determined, and it can be further determined whether frame extraction and image processing are required shot by shot, and finally a de-filled video can be obtained.

[0133] The following describes an apparatus embodiment of the present application, which can be used to execute the image processing method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the image processing method of the present application.

[0134] Fig.10 FIG. 4 is a block diagram of an image processing device according to an embodiment of the present application. Fig.10 As shown, the device 1000 includes at least the following parts.

[0135] An image acquisition unit 1010 is used to acquire an image to be processed, wherein the image to be processed includes a filled instance and a non-filled instance;

[0136] A feature extraction unit 1020 is used to extract a pixel feature map and a connection feature map of the image to be processed, wherein the pixel feature map is used to characterize whether a target pixel in the filled image belongs to a filled instance, and the connection feature map is used to characterize whether the target pixel and its neighboring pixels belong to the same instance;

[0137] The target detection frame acquisition unit 1030 is used to construct a connected domain according to the pixel feature map and the connection feature map to obtain a target detection frame corresponding to the non-filled instance;

[0138] The image capture unit 1040 is used to capture the image to be processed according to the target detection frame to obtain a de-filled image corresponding to the image to be processed.

[0139] Fig.11 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0140] It should be noted that Fig.11The computer system 1100 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0141] like Fig.11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1102 or the program loaded from the storage part 1108 to the random access memory (RAM) 1103, such as executing the method in the above embodiment. In the RAM 1103, various programs and data required for system operation are also stored. The CPU 1101, the ROM 1102 and the RAM 1103 are connected to each other through the bus 1104. The input / output (I / O) interface 1105 is also connected to the bus 1104.

[0142] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed so that a computer program read therefrom is installed into the storage section 1108 as needed.

[0143] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1109, and / or installed from a removable medium 1111. When the computer program is executed by a central processing unit (CPU) 1101, various functions defined in the system of the present application are executed.

[0144] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0145] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0147] As another aspect, the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method for verifying the video editing effect described in the above embodiment.

[0148] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method for verifying the video editing effect described in the above embodiment.

[0149] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0150] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0151] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0152] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be performed without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed, wherein the image to be processed includes a filled instance and a non-filled instance, the filled instance represents an image of a filled area in the image to be processed, the non-filled instance represents an image of a non-filled area in the image to be processed, the edge between the filled instance and the non-filled instance is significant, and the filled instance and the non-filled instance have a feature of obvious horizontal or vertical distribution, and the filled instance is a rectangle; Extracting a pixel feature map and a connection feature map of the image to be processed by a convolutional neural network, wherein the pixel feature map is used to characterize whether a target pixel in the image to be processed belongs to the filling instance, and the connection feature map is used to characterize whether the target pixel and its neighboring pixels belong to the same instance; The pixel feature map is classified to obtain a positive pixel set, in which the target pixel belongs to the filling instance, and the connection feature map is classified to obtain a positive connection set, in which the target pixel and the neighbor pixel belong to the same instance, and a connected domain of all pixels is constructed by merging the positive pixel set and the positive connection set, and a minimum enclosing regular rectangle is taken based on the connected domain to obtain a target detection frame; The image to be processed is intercepted according to the target detection frame to obtain a defilled image corresponding to the image to be processed.

2. The image processing method according to claim 1, characterized in that: The extracting the pixel feature map and the connection feature map of the image to be processed by the convolutional neural network includes: Performing multi-level convolution coding on the image to be processed to obtain a multi-level feature map; Performing a first convolution operation on the feature maps of the preset multiple levels with the convolution kernel of the first preset channel respectively, and performing feature fusion on the feature maps after the first convolution operation to obtain the pixel feature map; A second convolution operation is performed on the feature maps of the preset multiple levels respectively with the convolution kernel of the second preset channel, and feature fusion is performed on the feature maps after the second convolution operation to obtain the connection feature map.

3. The image processing method according to claim 2, characterized in that: The preset multiple levels of feature maps include a feature map of the last level among the multiple levels of feature maps, and the method further includes before performing the first convolution operation and the second convolution operation: Performing parallel sampling on the feature map of the last level using a first preset number of dilated convolutions to obtain a first preset number of sampled feature maps; The first preset number of sampled feature maps are cascaded to obtain an updated feature map of the final level.

4. The image processing method according to claim 1, characterized in that: The method further comprises: Extracting multiple frames of images to be processed from the video to be processed; Processing the multiple frames of images to be processed to obtain target detection frames of the multiple frames of images to be processed respectively; Determining the target detection frame of each frame image in the video to be processed according to the target detection frames of the multiple frames of images to be processed; Each frame image in the video to be processed is intercepted according to the target detection frame of each frame image in the video to be processed, so as to obtain a de-filled video corresponding to the video to be processed.

5. The image processing method according to claim 4, characterized in that: The step of extracting multiple frames of images to be processed from the video to be processed includes: If the duration of the video to be processed is greater than the first preset duration, intercepting a video of the second preset duration in the video to be processed to obtain an intercepted video; A plurality of frames of images are extracted at equal intervals from the captured video to obtain the plurality of frames of images to be processed.

6. The image processing method according to claim 4, characterized in that: The step of determining the target detection frame of each frame of the image in the video to be processed according to the target detection frames of the multiple frames of the image to be processed comprises: If the position coordinates of the target detection frames of at least two frames of the images to be processed are the same, determining the target detection frames of the at least two frames of the images to be processed as the confident target detection frames; According to the number of the confident target detection frames, the target detection frame of each frame image in the video is determined.

7. The image processing method according to claim 6, characterized in that: The step of determining the target detection frame of each frame image in the video according to the number of the confident target detection frames comprises: If the multiple frames of images to be processed include only one of the confident target detection frames, the confident target detection frame is determined as the target detection frame of each frame image in the video; If the multiple frames of images to be processed include at least two of the confident target detection frames, respectively intercepting the videos of the respective shots included in the video to be processed to obtain intercepted videos corresponding to the respective shots; Extracting multiple frames of images to be processed from the intercepted videos corresponding to the respective shots, to obtain images to be processed corresponding to the respective shots; Processing the to-be-processed images corresponding to the respective lenses to determine target detection frames of the to-be-processed images corresponding to the respective lenses; According to the target detection frames of the to-be-processed images corresponding to the respective shots, target detection frames of the respective frame images in the videos of the respective shots are determined.

8. An image processing device, characterized in that: The device comprises: An image acquisition unit is used to acquire an image to be processed, wherein the image to be processed includes a filled instance and a non-filled instance, the filled instance represents an image of a filled area in the image to be processed, and the non-filled instance represents an image of a non-filled area in the image to be processed, the edge between the filled instance and the non-filled instance is significant, and the filled instance and the non-filled instance have a feature of obvious horizontal or vertical distribution, and the filled instance is a rectangle; A feature extraction unit, used to extract a pixel feature map and a connection feature map of the image to be processed through a convolutional neural network, wherein the pixel feature map is used to characterize whether a target pixel in the image to be processed belongs to the filling instance, and the connection feature map is used to characterize whether the target pixel and its neighboring pixels belong to the same instance; a target detection frame acquisition unit, configured to classify the pixel feature map to obtain a positive pixel set, in which the target pixel belongs to the filling instance, and classify the connection feature map to obtain a positive connection set, in which the target pixel and the neighbor pixel belong to the same instance, and construct a connected domain of all pixels by merging the positive pixel set and the positive connection set, and obtain a target detection frame by taking a minimum enclosing regular rectangle based on the connected domain; The image capture unit is used to capture the image to be processed according to the target detection frame to obtain a defilled image corresponding to the image to be processed.

9. The image processing device according to claim 8, characterized in that: The feature extraction unit is configured as follows: Performing multi-level convolution coding on the image to be processed to obtain a multi-level feature map; Performing a first convolution operation on the feature maps of the preset multiple levels with the convolution kernel of the first preset channel respectively, and performing feature fusion on the feature maps after the first convolution operation to obtain the pixel feature map; A second convolution operation is performed on the feature maps of the preset multiple levels respectively with the convolution kernel of the second preset channel, and feature fusion is performed on the feature maps after the second convolution operation to obtain the connection feature map.

10. The image processing device according to claim 9, characterized in that: The preset multiple levels of feature maps include a feature map of the last level among the multiple levels of feature maps, and the device further includes: A dilated convolution unit, configured to perform parallel sampling on the feature map of the last level using a first preset number of dilated convolutions to obtain a first preset number of sampled feature maps; The atrous convolution unit is further used to cascade the first preset number of sampled feature maps to obtain an updated feature map of the final level.

11. The image processing device according to claim 8, characterized in that: The device also includes: An image extraction unit, used for extracting multiple frames of images to be processed from the video to be processed; The target detection frame acquisition unit is further used to process the multiple frames of images to be processed to obtain target detection frames of the multiple frames of images to be processed respectively; The target detection frame acquisition unit is further used to determine the target detection frame of each frame image in the to-be-processed video according to the target detection frames of the multiple frames of images to be processed; The image capture unit is further used to capture each frame of the video to be processed according to the target detection frame of each frame of the video to be processed, so as to obtain a de-filled video corresponding to the video to be processed.

12. The image processing device according to claim 11, characterized in that: The image extraction unit is configured as follows: If the duration of the video to be processed is greater than the first preset duration, intercepting a video of the second preset duration in the video to be processed to obtain an intercepted video; A plurality of frames of images are extracted at equal intervals from the captured video to obtain the plurality of frames of images to be processed.

13. The image processing device according to claim 11, characterized in that: The target detection frame acquisition unit is configured as follows: If the position coordinates of the target detection frames of at least two frames of the images to be processed are the same, determining the target detection frames of the at least two frames of the images to be processed as the confident target detection frames; According to the number of the confident target detection frames, the target detection frame of each frame image in the video is determined.

14. The image processing device according to claim 13, characterized in that: The target detection frame acquisition unit is further configured to: If the multiple frames of images to be processed include only one of the confident target detection frames, the confident target detection frame is determined as the target detection frame of each frame image in the video; If the multiple frames of images to be processed include at least two of the confident target detection frames, respectively intercepting the videos of the respective shots included in the video to be processed to obtain intercepted videos corresponding to the respective shots; Extracting multiple frames of images to be processed from the intercepted videos corresponding to the respective shots, to obtain images to be processed corresponding to the respective shots; Processing the to-be-processed images corresponding to the respective lenses to determine target detection frames of the to-be-processed images corresponding to the respective lenses; According to the target detection frames of the to-be-processed images corresponding to the respective shots, target detection frames of the respective frame images in the videos of the respective shots are determined.

15. An electronic device, comprising: one or more processors; A storage device for storing one or more programs, so that when the one or more programs are executed by the electronic device, the method described in any one of claims 1 to 7 is implemented.

16. A computer readable medium, characterized in that A computer program is stored thereon, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images