Image processing method and device, computer equipment, readable storage medium and product

Abstract

The embodiment of the invention provides an image processing method and device, computer equipment, a readable storage medium and a product. The method comprises the following steps: acquiring a first foreground image and a first background image; determining a first focus of the first foreground image and a second focus of the first background image; sampling in the first motion speed range and the first motion angular speed range to determine first change line track information of the first focus; sampling in a second motion speed range and a second motion angular speed range to determine second change line track information of a second focus; updating first position information of a first pixel in the first foreground image at each moment according to the first change line track information; updating second position information of a second pixel in the first background image at each moment according to the second change line track information; and combining the updated first foreground image and the first background image at the same moment to obtain a second image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, an image processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Deep learning performs well in visual tasks such as image classification and instance segmentation, but it relies on a large amount of training data. In real-world applications, image data collection is costly and poses privacy and legal risks. Data enhancement has become an important means to solve the problem of insufficient data.

[0003] General data enhancement methods transform at the image level, the difference is not obvious, and it is difficult to change the video motion in the time dimension; the specific acquisition cost is high and the scene is difficult to keep consistent. Summary of the invention

[0004] An embodiment of the present disclosure provides an image processing method, which includes: acquiring a first foreground image and a first background image; determining a first focus of the first foreground image and a second focus of the first background image; determining first change line trajectory information of the first focus by sampling within a first motion speed range and a first motion angular velocity range; determining second change line trajectory information of the second focus by sampling within a second motion speed range and a second motion angular velocity range; updating first position information of a first pixel in the first foreground image at each moment according to the first change line trajectory information; updating second position information of a second pixel in the first background image at each moment according to the second change line trajectory information; and combining the first foreground image and the first background image at the same moment after the update to obtain a second image.

[0005] An embodiment of the present disclosure provides an image processing device, which includes: an acquisition unit, used to acquire a first foreground image and a first background image; a determination unit, used to determine a first focus of the first foreground image and a second focus of the first background image; a sampling unit, used to determine first change line trajectory information of the first focus by sampling within a first motion speed range and a first motion angular velocity range; the sampling unit is also used to determine second change line trajectory information of the second focus by sampling within a second motion speed range and a second motion angular velocity range; an updating unit, used to update first position information of a first pixel in the first foreground image at each moment according to the first change line trajectory information; the updating unit is also used to update second position information of a second pixel in the first background image at each moment according to the second change line trajectory information; a combining unit, used to combine the first foreground image and the first background image at the same moment after the update to obtain a second image.

[0006] An embodiment of the present disclosure provides a computer device, including a processor, a memory, and an input / output interface; the processor is connected to the memory and the input / output interface, respectively, wherein the input / output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device including the processor executes the image processing method in any embodiment of the present disclosure.

[0007] An embodiment of the present disclosure provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the image processing method in any embodiment of the present disclosure.

[0008] The embodiments of the present disclosure provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional modes in any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 It is a network interaction architecture diagram of an image processing method provided by an embodiment of the present disclosure.

[0011] Figure 2 It is a flowchart of an image processing method provided by an embodiment of the present disclosure.

[0012] Figure 3 It is a schematic diagram of image data enhancement provided by an embodiment of the present disclosure.

[0013] Figure 4 It is a schematic diagram of the overall flow of an image processing method provided by an embodiment of the present disclosure.

[0014] Figure 5 It is a schematic diagram of an image processing method provided by an embodiment of the present disclosure.

[0015] Figure 6 is a schematic diagram of an image processing device provided by an embodiment of the present disclosure.

[0016] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0018] Figure 1 is a network interaction architecture diagram of an image processing method provided by an embodiment of the present disclosure. In the embodiment of the present disclosure, see Figure 1 , Figure 1 1 is a network interaction architecture diagram of an image processing method provided by an embodiment of the present disclosure, and the embodiment of the present disclosure can be implemented by a terminal and / or a server. The terminal can obtain data from a computer device 101 and display the data, wherein the computer device 101 can exchange data with the terminal, and the computer device 101 can be a server where an application is located, or can belong to a terminal (i.e., the background of the terminal), etc., without limitation here.

[0019] Among them, the terminal can be terminal 102a, terminal 102b or terminal 102c, etc., and the embodiment of the present disclosure can be implemented by any one of the terminals 102a, terminal 102b or terminal 102c, etc. Specifically, terminal 102b is taken as an example. The computer device 101 is used to obtain a first foreground image and a first background image; determine a first focus of the first foreground image and a second focus of the first background image; determine the first change line trajectory information of the first focus by sampling within a first motion speed range and a first motion angular velocity range; determine the second change line trajectory information of the second focus by sampling within a second motion speed range and a second motion angular velocity range; update the first position information of the first pixel in the first foreground image at each time according to the first change line trajectory information; update the second position information of the second pixel in the first background image at each time according to the second change line trajectory information; combine the first foreground image and the first background image at the same time after the update to obtain a second image. The computer device 101 sends the obtained second image or the second video including the second image to terminal 102b. Terminal 102b displays the received second image or second video. The computer device 101 may be a server corresponding to the target application, or may be a background of the terminal 102b, etc., which is not limited here.

[0020] The terminal may be a mobile phone (such as terminal 102c) or a laptop (such as terminal 102b), or a playback device in a vehicle (such as terminal 102a), etc., without limitation. The terminal 102a may be considered as a playback device in the vehicle 103, and the target application may be displayed in the terminal 102a. Figure 1 The terminals in the example are only some of the devices. In the present disclosure, the terminals are not limited to the Figure 1 The target application in the present disclosure may be any application program capable of displaying multimedia information.

[0021] It is understandable that the terminal mentioned in the embodiments of the present disclosure may be a computer device, and the computer device in the embodiments of the present disclosure includes but is not limited to a terminal or a server. In other words, the computer device may be a server or a terminal, or a system consisting of a server and a terminal. Among them, the terminal mentioned above may be an electronic device, including but not limited to a mobile phone, a tablet computer, a desktop computer, a laptop computer, a PDA, a vehicle-mounted device, an augmented reality / virtual reality (AR / VR) device, a helmet display, a smart TV, a wearable device, a smart speaker, a digital camera, a camera, and other mobile Internet devices (mobile internet device, MID) with network access capabilities, or terminals in scenarios such as trains, ships, and flights.

[0022] Among them, the servers mentioned above can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0023] Optionally, the data involved in the embodiments of the present disclosure may be stored in a computer device, or may be stored based on cloud storage technology, or may be stored through blockchain technology, without limitation herein.

[0024] Figure 2 It is a flowchart of an image processing method provided by an embodiment of the present disclosure. Figure 2 The method provided in the embodiment can be executed by any computer device. Figure 2 As shown, the method provided by the embodiment of the present disclosure may include the following steps.

[0025] In S210, a first foreground image and a first background image are acquired.

[0026] In an exemplary embodiment, obtaining a first foreground image and a first background image includes: obtaining a first video; extracting an object contained in the first video through a semantic segmentation model to obtain first background data and first foreground data containing the object; determining the first foreground image containing the object from the first foreground data; and completing the first background data by generating a network model to obtain the first background image.

[0027] In the embodiment of the present disclosure, the first image may be any original image to be subjected to data enhancement processing. The first image may be from a first video. For example, the first image may be an image that includes at least one object in the first video and contains a relatively complete image of the object and a relatively clear image. The first video may be any video to be subjected to data enhancement processing.

[0028] It should be noted that the first image of the first foreground image and the first image of the first background image in the disclosed embodiment may be the same or different. For example, if the first video contains a cat and a dog, wherein the first frame image contains the cat and the fifth frame image contains the dog, then the first foreground image may be the first frame image containing the cat and the fifth frame image containing the dog. The first background image may be an image after the cat in the first frame image is cut out, or an image after the dog in the fifth frame image is cut out, or the background of any frame image extracted from the first video.

[0029] The first foreground image in the disclosed embodiment refers to an element or part of an image or a scene displayed by the image that is closer to the observer. Due to the closer distance, the foreground is more prominent and clear in visual perception. It can be an object, a person, an animal, a detail, or any element or object that visually attracts the observer's attention first in the image. The first foreground image may include one or more objects.

[0030] The first background data in the disclosed embodiment refers to the elements or parts of the image that are behind the foreground and are farther from the observer. The background is not as prominent as the foreground visually, and it provides environment and context for the foreground. The background can be a natural environment, a building, interior decoration, or any other element that constitutes a scene.

[0031] In some embodiments, a semantic segmentation model may be used to perform semantic segmentation on each image in the first video to identify, locate, and segment each object contained therein, and to separate the foreground from the background.

[0032] In the disclosed embodiments, the semantic segmentation model refers to a deep learning model or machine learning model that can classify each pixel in an image into a corresponding category, that is, implement pixel-level classification. For example, any one or more of FCN (Fully Convolutional Networks), SegNet, PSPNet (Pyramid Scene Parsing Network), DeepLab, etc. can be used.

[0033] In the disclosed embodiment, after the foreground is removed from the first image, at least part of the first background data is blank, and this blank area can be filled, and the filled image is called the first background image.

[0034] In some embodiments, the first background data can be completed by a generative network model to generate a first background image. The generative network is part of a generative adversarial network. The generative adversarial network includes a generative network model and a discriminative network model. The generative network model in the disclosed embodiment refers to a network suitable for image completion tasks, which takes the original occluded image (i.e., the first image with the foreground removed, i.e., the first background data) as input and outputs the completed image, i.e., the first background image. It should be noted that the completion here is not to complete the foreground that was originally deducted, but to fill in the blank part of the image with the foreground removed.

[0035] In the disclosed embodiment, a generative adversarial network is first trained based on training samples, and then the generative network model is used to complete the image.

[0036] In S220, a first focus of the first foreground image and a second focus of the first background image are determined.

[0037] In the embodiments of the present disclosure, the first focus refers to, when assuming that the first foreground image is captured by a shooting device (such as a camera, a camera lens, etc.), selecting a point from the first foreground image as the first focus, and adjusting the focus ring or focus mode of the shooting device to ensure that the first focus is clear. In the embodiments of the present disclosure, the coordinate center of the first foreground image can be assumed to be the first focus, but the present disclosure is not limited thereto.

[0038] The second focus refers to, when the first background image is captured by a shooting device (such as a camera, a camera lens, etc.), selecting a point from the first background image as the second focus, and adjusting the focus ring or focus mode of the shooting device to ensure that the second focus is clear. In the embodiment of the present disclosure, the coordinate center of the first background image can be assumed to be the second focus, but the present disclosure is not limited to this.

[0039] In S230, first change line trajectory information of the first focus is determined by sampling within a first movement speed range and a first movement angular velocity range.

[0040] In the embodiment of the present disclosure, assuming that the coordinate position of the first focus is the starting point P1 (assuming its coordinates are (x0, y0)), random sampling of the first motion speed v and the first motion angular velocity a of the first focus at each moment t is performed within the first motion speed range and the first motion angular velocity range, respectively, so as to determine the first motion speed v and the first motion angular velocity a of the first focus at each moment t. After determining the first motion speed v and the first motion angular velocity a of the first focus at each moment t, the first change line trajectory information of the first focus can be determined.

[0041] In an exemplary embodiment, first change line trajectory information of the first focus is determined by sampling within a first motion speed range and a first motion angular velocity range, including: determining a predetermined number of first time intervals; within the first time interval, sampling the first motion speed range and the first motion angular velocity range to determine a first sampled lens motion trajectory of the first focus within the first time interval; mapping the first sampled lens motion trajectory to an image space to obtain a first lens motion trajectory in the image space; within the first time interval, sampling a first starting point and a first end point of the first lens motion trajectory at angular velocity change intervals to obtain first sub-change line trajectory information of the first focus within the first time interval; and splicing the predetermined number of first sub-change line trajectory information to obtain the first change line trajectory information.

[0042] In the disclosed embodiment, the predetermined number can be determined based on the number of frames contained in the last synthesized second video and the total duration of the second video. For example, if the second video contains 10 frames and the total duration of the second video is 100 seconds, it can be divided into 10 10s, that is, a second image is generated every 10 seconds, and then the 10 second images are spliced ​​in chronological order to generate the second video. The first change line trajectory information generated in each first time interval is called the first sub-change line trajectory information.

[0043] In an exemplary embodiment, when sampling is performed within the first range of motion speed and the first range of motion angular speed, the first motion angular speed is fixed within the first time interval; or the first motion angular speed is dynamically changed.

[0044] In S240, first position information of a first pixel in the first foreground image at each moment is updated according to the first change line trajectory information.

[0045] In S250, second change line trajectory information of the second focus is determined by sampling within a second movement speed range and a second movement angular velocity range.

[0046] In an exemplary embodiment, second change line trajectory information of the second focus is determined by sampling within a second motion speed range and a second motion angular velocity range, including: within the first time interval, sampling the second motion speed range and the second motion angular velocity range to determine the second sampled lens motion trajectory of the second focus within the first time interval; mapping the second sampled lens motion trajectory to the image space to obtain the second lens motion trajectory in the image space; within the first time interval, sampling the second starting point and the second end point of the second lens motion trajectory at angular velocity change intervals to obtain second sub-change line trajectory information of the second focus within the first time interval; splicing the predetermined number of second sub-change line trajectory information to obtain the second change line trajectory information.

[0047] In an exemplary embodiment, the first range of motion speeds is different from the second range of motion speeds; and / or the first range of motion angular speeds is different from the second range of motion angular speeds. At the same time, the first motion speed sampled or adopted is different from the second motion speed, and the first motion angular speed is different from the second motion angular speed.

[0048] In the disclosed embodiment, the data enhancement of the image or video is performed by simulating a dynamic trajectory. In the process of generating the dynamic trajectory, the foreground data (e.g., the first foreground image, such as the main character or object in the first video) and the background data (e.g., the first background image, such as the background environment in the image) are processed separately, and are sampled in different motion speed ranges and motion angular velocity ranges, respectively, and the sampling lens motion trajectories of the foreground and background data (e.g., the first sampling lens motion trajectory and the second sampling lens motion trajectory) are respectively constructed, and are respectively mapped back to the image space for angular velocity change interval sampling, and the first change line trajectory information and the second change line trajectory information are respectively obtained. Since the foreground and the background change at different motion speeds and motion angular velocities at the same time, a more realistic lens motion effect can be simulated.

[0049] In the disclosed embodiment, different motion speed ranges and motion angular velocity ranges are set for the foreground and background, respectively, and then sampling is performed in the first motion speed range and the first motion angle range, and the second motion speed range and the second motion angle range, respectively. This is like setting a set of "motion rules" for the foreground and background respectively, and then randomly selecting their motion methods according to these rules. Then, based on the motion information obtained from these samples, the motion trajectory of the sampling lens (i.e., the motion trajectory of the first sampling lens and the motion trajectory of the second sampling lens) is constructed. The motion trajectories of the two sampling lenses are like the paths that the cameras move when shooting the first foreground image and the first background image.

[0050] Since the motion trajectories of the two sampling lenses are constructed in an abstract space (called "voxel space"), in order to apply these two trajectories to the actual image, they are mapped back to the image space (height, width), that is, a two-dimensional plane image space including height and width. This is like projecting a three-dimensional model onto a two-dimensional plane.

[0051] In the image space, the angular velocities of the start and end points of the lens are sampled at varying intervals at specified time intervals (i.e., the first time interval described above, for example, every 10 seconds), which is like recording the motion state of the camera at each key time point.

[0052] In the disclosed embodiment, a specified time interval is set, for example, every 10 seconds. This means that the motion state of the lens is recorded or sampled once every 10 seconds. In each such time interval, special attention is paid to the angular velocity of the starting point (including the first starting point and the second starting point) and the end point (including the first end point and the second end point) of the lens. The angular velocity describes the speed of the lens rotation. By sampling the angular velocity of the starting point and the end point at a variable interval, that is, sampling the angular velocity of the lens at different time points to obtain its change process, it is possible to capture how the lens starts to rotate, how much it rotates, and how it stops rotating during this period of time, thereby obtaining the angular velocity change information during the movement of the lens. The specific sampling method of the variable interval can be designed according to the actual scene. Through these sampling points, the motion trajectory of the lens during this period can be reconstructed, thereby generating more realistic and complex video data. This step is to simulate the complexity and dynamics of the real lens movement, and ensure that the generated video sequence (such as the second video) is closer to the real scene.

[0053] In S260, the second position information of the second pixel in the first background image at each moment is updated according to the second change line trajectory information.

[0054] In S270, the first foreground image and the first background image at the same time after updating are combined to obtain a second image.

[0055] In the disclosed embodiment, according to the motion states obtained by these samplings, the change line trajectory information of the lens in the whole motion process (including the first change line trajectory information and the second change line trajectory information) can be obtained. The first change line trajectory information and the second change line trajectory information can be used to update the position of each pixel point (referred to as the first pixel and the second pixel respectively for the sake of distinction) in the first foreground image and the first background image, so as to be used for the subsequent generation of a new image (i.e., the second image) that simulates the motion of the lens.

[0056] In the disclosed embodiment, each pixel point of the first foreground image is updated according to the first change line trajectory information to obtain new coordinates at each moment; each pixel point of the first background image is updated according to the second change line trajectory information to obtain new coordinates at each moment; then the first foreground image and the first background image at the same moment are masked and combined to generate a new second image, and the second images at each moment are spliced ​​or combined in chronological order to generate a new mixed video sequence, i.e., the second video.

[0057] In an exemplary embodiment, the first foreground image and the first background image at the same time after being updated are combined to obtain a second image, including: respectively combining the first foreground image at the same time after being updated and containing different objects and different first background images to obtain the second image.

[0058] The embodiment of the present disclosure can combine different first foreground images and first background images, so that the generated new second image can contain objects that are not included in the original first image.

[0059] In an exemplary embodiment, the method provided by the embodiment of the present disclosure further includes: splicing a plurality of the second images to obtain a second video.

[0060] The image processing method provided by the embodiment of the present disclosure is a method for enhancing video or image data by mixing foreground and background and generating random lens trajectories.

[0061] The disclosed embodiment is based on the principle of multi-target two-dimensional transformation, and obtains the affine transformation matrix of the lens movement trajectory by combining two-dimensional data transformation methods such as translation, rotation, and expansion. In the disclosed embodiment, there is no need to calculate the specific affine transformation matrix. For multiple targets or objects in the first video, for example, 5 targets (people, dogs, cats, etc.) are selected, and the 5 targets are used as the first focus respectively to simulate the lens movement trajectory (i.e., the first lens movement trajectory), and the lens movement trajectory of the background is simulated (i.e., the second lens movement trajectory), and then at the same time t, the 5 targets are combined with the background to form a new picture (i.e., the second image). The disclosed embodiment forms a new motion video sequence by setting different motion trajectories for the foreground and background (the motion speed and motion angular velocity of the foreground and background at the same time are different) and combining them, so as to achieve the purpose of data enhancement.

[0062] In recent years, deep learning has achieved outstanding performance in visual tasks such as image classification and instance segmentation, which usually rely on the use of a large amount of training data. However, in many real application scenarios, the cost of collecting image data is high and there are legal risks such as privacy and human rights, so it is usually difficult to obtain sufficient image data. Image enhancement can solve the problem of insufficient data. Data enhancement has become an important part of deep learning models. Data enhancement or image enhancement is a technology in machine learning, especially in the field of deep learning, which aims to improve the performance of the model by expanding the training data set, improving the generalization ability of the model, and reducing overfitting.

[0063] In an exemplary embodiment, the method provided by the embodiment of the present disclosure also includes: training a deep learning model using the second image.

[0064] The disclosed embodiment sets the motion rules of the foreground and background, simulates the motion trajectory of the camera, and applies these motion effects to the image, thereby generating more realistic and diverse data to enhance the training data set. The enhanced training data set is then used to train the deep learning model, which can expand the universality of the deep learning model and improve the performance of the deep learning model. The disclosure does not limit the type of deep learning model.

[0065] Compared with the data enhancement methods in the related art, the method provided by the embodiment of the present disclosure enhances the video data by mixing the foreground and background and generating random trajectories of the lens. The formed video sequence (i.e., the second video) is more complex and has a smaller gap with the video sequence of complex motion in real situations, which helps to expand the amount of training data without increasing the cost of data collection and increase the applicability of the visual model. The embodiment of the present disclosure provides a method for increasing data diversity as much as possible on the basis of limiting costs, which can expand the universality of the model.

[0066] The disclosed embodiments, by generating more complex and realistic motion video sequences, can, on the one hand, train more robust and adaptable deep learning models. These models can better handle complex motion situations in actual scenes. On the other hand, no additional data collection and annotation work is required, thereby reducing costs. At the same time, by generating a large number of new video sequences, the size of the training data set can be significantly expanded, which helps to improve the generalization ability and accuracy of the model.

[0067] Figure 3 is a schematic diagram of an image data enhancement provided by an embodiment of the present disclosure. Data enhancement is the process of artificially increasing the amount of data by generating new data points based on the original data. It includes perturbing the data in different directions or using deep learning to generate new data points in the original data space.

[0068] like Figure 3 As shown in the figure, the left side is the original input, and the right side is the enhanced output after data enhancement. The data enhancement methods include rotation (rotating the image between 0-360 degrees), translation (obtaining a new image by moving up, down, left, and right), shearing (moving in different directions along the width and height directions) and flipping (moving each pixel value to a symmetrical position along the central axis of the image).

[0069] The advantages of the above method are that the calculation method is simple and easy to implement; it is suitable for simple scenes such as slow motion and fixed shots. The disadvantages are that there is a gap with the complex shooting scenes and lens movement scenes in the real world; and the features of the enhanced data and the existing data are highly overlapped, and the difference is not obvious.

[0070] Since data enhancement is generally processed based on collected data, the real video data collected is mostly simple two-dimensional motion patterns. Deep learning models trained based on this data, such as visual models (deep learning models used to process videos or images), are difficult to obtain obvious reasoning effects in complex scenes, such as high-speed motion and severe shaking scenes. The specific collection of corresponding motion data has a high collection cost on the one hand, and it is difficult to maintain consistency in the collection scene with other data on the other hand. The general data enhancement method only transforms at the image level. The features of the enhanced data and the existing data are highly overlapped, and the difference is not obvious. It is difficult to change the entire video motion in the time dimension.

[0071] The method provided by the embodiment of the present disclosure simulates the motion of the video, and the camera has a motion speed and a motion angular velocity (rotation) in the time dimension. Therefore, the method provided by the embodiment of the present disclosure can increase data diversity as much as possible on the basis of limiting costs and expand the universality of the model.

[0072] Data enhancement in the disclosed embodiments refers to the process of generating new data points based on foreground and background data to increase the amount of data, and generating a picture that is as close to the actual situation as possible by simulating the actual motion trajectory of the camera.

[0073] Figure 4 FIG. 1 is a schematic diagram of the overall process of an image processing method provided by an embodiment of the present disclosure. Figure 4 As shown, the image processing method provided by the embodiment of the present disclosure is a data enhancement method, which can be divided into foreground and background segmentation, completion, dynamic trajectory generation and mask combination.

[0074] refer to Figure 4 The specific process of data enhancement proposed in the embodiment of the present disclosure is as follows:

[0075] Step 1: The collected training samples (ie, images, including the first image) are semantically segmented to separate the foreground and the background, and the foreground data and the background data (also referred to as background data) are extracted.

[0076] In image or video processing, foreground and background segmentation refers to separating the foreground (object or region of interest) from the background in an image or video. Foreground and background segmentation techniques include threshold-based methods, edge detection-based methods, region-based methods, and deep learning-based methods (such as semantic segmentation).

[0077] In an exemplary embodiment, the semantic segmentation model used is U-net, which is pre-trained based on the PASCAL dataset. However, the present disclosure is not limited thereto, and other semantic segmentation models may be used for semantic segmentation.

[0078] In the disclosed embodiment, the foreground sequence is composed of several (two or more) foreground video segments obtained through semantic segmentation, and foreground samples of a specific category are obtained through classified sampling as foreground sequence blocks.

[0079] For example, suppose that the first video is semantically segmented, and each target in the first video, such as a cat, a dog, a person, etc., is extracted as foreground data. Assuming that the cat exists within 0-15s of the first video, the 0-15s are sampled from the first video as the foreground sequence of the cat. In actual scenarios, the 15s video clip may not be used in full, so further sampling can be performed, such as sampling a frame in which the picture of the cat is clear and complete as a foreground sequence block, that is, as the first foreground image of the cat. It can be understood that the step of sampling the foreground sequence is not necessary and can be determined according to actual conditions. Assuming that the dog exists within 5-20s of the first video, the 5-20s are sampled from the first video as the foreground sequence of the dog. In actual scenarios, the 15s video clip may not be used in full, so further sampling can be performed, such as sampling a frame in which the picture of the dog is clear and complete as a foreground sequence block, that is, as the first foreground image of the dog. Assuming that the person exists within 0-20s of the first video, the 0-20s are sampled from the first video as the foreground sequence of the person. In actual scenarios, the 20-second video clip may not be fully used, so further sampling may be performed, for example, sampling a frame in which a person's image is clear and complete as a foreground sequence block, that is, as the first foreground image of the person.

[0080] Step 2: The background data is completed by generating a network model to obtain completed background data, namely, completed background data. The completed background data is sampled to obtain a background sequence, which includes the first background image.

[0081] In the disclosed embodiment, completion refers to filling or repairing the missing part in the image. Completion techniques include interpolation-based methods, image restoration-based methods, and deep learning-based methods (such as generative adversarial networks).

[0082] The generative network model in the embodiment of the present disclosure may be a network based on ESRGAN (super-resolution generative adversarial network) and multi-scale feature fusion, but the present disclosure is not limited thereto. In some embodiments, the generative network model is pre-trained based on the ImageNet dataset.

[0083] Step 3. The data enters the dynamic trajectory generation module, and the foreground and background data are sampled in different motion speed ranges and motion angular velocity ranges. Exemplarily, the image focus selects any point in the voxel space with a scale of 1. In some embodiments, the sampled lens motion trajectory is constructed within a specified time interval such as 10 seconds.

[0084] The dynamic trajectory generation in the embodiment of the present disclosure refers to generating new motion trajectories based on the foreground sequence blocks and the background sequence blocks, respectively. In data enhancement, the dynamic trajectory generation can be used to generate new training samples (such as the second image or the second video mentioned above).

[0085] The voxel space in the embodiment of the present disclosure may be a three-dimensional space or a two-dimensional space. When it is a two-dimensional space, the voxel may also be understood as a pixel. That is, a focus (i.e., a first focus and a second focus) is selected from the image (including the first foreground image and the first background image). For example, assuming that the camera takes pictures at the center point of the image by default, the center point may be selected as the focus.

[0086] The disclosed embodiment can prevent the foreground and background from having too large a difference in motion by constructing a sampled lens motion trajectory (including a first sampled lens motion trajectory and a second sampled lens motion trajectory) at a specified time interval such as 10 seconds. For example, assuming that a 60-second second video is generated, it is actually divided into 6 segments, each 10 seconds long, and a motion trajectory of every 10 seconds is generated, which is then spliced ​​to make the motion trajectory change more smoothly. Figure 5 The sampled camera / lens motion trajectory 502 of the middle camera 501 is actually composed of two segments (a first segment 502-1 and a second segment 502-2) spliced ​​together.

[0087] How to divide the length of each segment depends on the total duration of the second video and the number of frames to be synthesized. For example, if you want to synthesize a 100s video, you actually only need 10 frames, which can be divided into 10s intervals.

[0088] Step 4. Map the acquired sampled lens motion trajectory in the voxel space back to the image space (height, width), perform interval sampling of the angular velocity transformation of the starting point (including the first starting point and the second starting point) and the end point (including the first end point and the second end point) at a specified time interval, and obtain the first change line trajectory information and the second change line trajectory information.

[0089] In the embodiment of the present disclosure, the first motion speed and the first motion angular speed, and the second motion speed and the second motion angular speed of the camera may be sampled according to any distribution. Figure 4 The normal distribution is taken as an example, but the present disclosure is not limited to this, and any other distribution such as Poisson distribution can also be adopted. When the starting coordinates (assuming it is P1) of the focus (including the first focus and the second focus), the movement speed (including the first movement speed and the second movement speed) and the movement angular velocity (including the first movement angular velocity and the second movement angular velocity) are known, its movement trajectory can be known, and according to the movement trajectory, the coordinates of the focus at any time can be known. In a similar way, the coordinates of any pixel point in the first foreground image and the first background image at any time t can be obtained, and then the coordinates of all pixel points in the first foreground image and the first background image at the same time are merged to obtain the image at time t, that is, the new picture it at any time t, that is, the second image.

[0090] In the embodiment of the present disclosure, if the purpose is to obtain a relatively smooth motion trajectory, the angular velocity of movement may be fixed within the time interval, for example, 10 seconds (the first angular velocity of movement and / or the second angular velocity of movement may be fixed); if the purpose is to obtain a fast motion trajectory, the angular velocity of movement may be dynamically changed within the time interval, for example, 10 seconds (the first angular velocity of movement and / or the second angular velocity of movement may be dynamically changed).

[0091] Step 5. The coordinates (X, Y) of each pixel point i in the first foreground image and the first background image are updated according to the first change line trajectory information and the second change line trajectory information, respectively, to obtain the new coordinates i (Xt, Yt) at any time t. By sorting out the coordinates of each pixel point at time t, the new image it at time t can be obtained.

[0092] The foreground and background are processed according to steps 3-5 to obtain the transformed foreground sequence block (including the updated first foreground image) and the transformed background sequence (including the updated first background image). The two are masked and combined to generate a new mixed video sequence, i.e., the second video.

[0093] In the disclosed embodiments, mask combination refers to superimposing or combining different images or data layers through masks. In data enhancement, mask combination can be used to generate new samples with different features or attributes. For example, the foreground of one image can be combined with the background of another image.

[0094] The method provided in the embodiment of the present disclosure can be used to generate mixed video data containing high-dimensional motion information, thereby expanding the amount of training data without increasing data costs and increasing the applicability of the visual model.

[0095] Figure 5 It is a schematic diagram of an image processing method provided by an embodiment of the present disclosure.

[0096] Camera, motion trajectory, scene, mixed video sequence, such as Figure 5 As shown, taking the COCO dataset as an example, through the data enhancement method provided above, a combined video sequence 504 can be generated based on the original first image 503, and the combined video sequence 504 includes multiple frames of second images at T=0, T=t1, T=t2, T=t3, and T=t4, that is, video sequence data containing complex manifold motion can be generated. In other embodiments, it is also possible to add targets that were not originally there in the images at T=0, T=t1, T=t2, T=t3, and T=t4, such as adding a cat. This is because different foregrounds and backgrounds are combined.

[0097] The method provided by the embodiment of the present disclosure is a data enhancement method based on the combination of foreground and background and the simulation of random trajectory of the lens. It relates to the field of artificial intelligence and computer vision technology. The method provided by the embodiment of the present disclosure performs data increment by simulating the motion trajectory of the lens / camera and fusing the foreground and background data, so as to solve the problem that the amount of data is insufficient or the small sample data cannot fit the real scene under the condition of limited acquisition cost. The formed mixed foreground and background video sequence will participate in the training process of the subsequent task network as new training data, thereby increasing the performance effect of the visual task. On the one hand, the diversity of the available data set is increased without increasing the cost of additional data acquisition. On the other hand, the mixed video sequence that meets different actual scenes can be generated by controlling the sampling range of the motion speed and the motion angular velocity. In addition, the motion speed construction method of the foreground and background difference can simulate the effect of lens shaking. Dynamic trajectory generation and foreground and background fusion simulate complex motion, so that the generated mixed video sequence is closer to reality, and the universality of training data and the robustness of the model are improved.

[0098] The hybrid synthesis method provided by the embodiment of the present disclosure will not increase the cost of additional data collection. The combination of different foregrounds and backgrounds can form data with sufficient diversity. This method can fit different lens motion scenes such as idling, medium speed, and high speed. This method can improve the universality of training data and the robustness of the model. Dynamic trajectory generation and foreground and background fusion are used to simulate complex motion. The separate simulation of foreground and background makes the generated hybrid video sequence closer to reality. The problem of high cost of image data collection and difficulty in obtaining sufficient data is solved.

[0099] Figure 6 is a schematic diagram of an image processing device provided by an embodiment of the present disclosure. Figure 6 As shown, the image processing device 600 provided by the embodiment of the present disclosure includes an acquisition unit 610, a determination unit 620, a sampling unit 630, an updating unit 640 and a combining unit 650.

[0100] The acquisition unit 610 is used to acquire a first foreground image and a first background image.

[0101] The determination unit 620 is configured to determine a first focus of the first foreground image and a second focus of the first background image.

[0102] The sampling unit 630 is used to determine the first change line trajectory information of the first focus by sampling within a first movement speed range and a first movement angular velocity range.

[0103] The sampling unit 630 is further configured to determine second change line trajectory information of the second focus by performing sampling within a second motion speed range and a second motion angular velocity range.

[0104] The updating unit 640 is used to update the first position information of the first pixel in the first foreground image at each time according to the first change line trajectory information.

[0105] The updating unit 640 is further configured to update the second position information of the second pixel in the first background image at each time point according to the second change line trajectory information.

[0106] The combining unit 650 is used to combine the first foreground image and the first background image at the same time after updating to obtain a second image.

[0107] The image processing apparatus may be a computer program (including program code, etc.) running in a computer device. For example, the image processing apparatus may be an application software. The apparatus may be used to execute corresponding steps in the method provided in the embodiment of the present disclosure.

[0108] Figure 6 For other contents of the embodiment, reference may be made to the other embodiments mentioned above.

[0109] Figure 7 Schematic diagram of a computer device provided by an embodiment of the present disclosure. Figure 7 , Figure 7 Schematic diagram of a computer device provided by an embodiment of the present disclosure. Figure 7 As shown, the computer device in the embodiment of the present disclosure may include: one or more processors 701, a memory 702, and an input-output interface 703. The processor 701, the memory 702, and the input-output interface 703 are connected via a bus 704. The memory 702 is used to store a computer program, which includes program instructions. The input-output interface 703 is used to receive data and output data, such as for data interaction between a host machine and a computer device, or for data interaction between various virtual machines in a host machine; the processor 701 is used to execute the program instructions stored in the memory 702.

[0110] The processor 701 may execute the method provided in any embodiment of the present disclosure.

[0111] In some feasible implementations, the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0112] The memory 702 may include a read-only memory and a random access memory, and provide instructions and data to the processor 701 and the input / output interface 703. A portion of the memory 702 may also include a non-volatile random access memory. For example, the memory 702 may also store information about the device type.

[0113] In a specific implementation, the computer device can execute the implementation methods provided by the various steps in any of the above method embodiments through its built-in functional modules. For details, please refer to the implementation methods provided by the various steps in the figure shown in the above method embodiments, which will not be repeated here.

[0114] The embodiments of the present disclosure provide a computer device, including a processor, an input / output interface, and a memory. The processor obtains a computer program in the memory to execute each step of the method shown in any of the above embodiments.

[0115] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by the processor and executing the image processing method provided by each step in any of the above embodiments. For details, please refer to the implementation method provided by each step in any of the above embodiments, which will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in the present disclosure, please refer to the description of the method embodiment of the present disclosure. As an example, the computer program can be deployed to be executed on a computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0116] The computer-readable storage medium may be the image processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0117] The present disclosure also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in any of the above-mentioned optional modes.

[0118] The terms "first", "second", etc. in the specification, claims, and drawings of the embodiments of the present disclosure are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other step units inherent to these processes, methods, devices, products, or equipment.

[0119] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0120] The method and related apparatus provided by the embodiments of the present disclosure are described with reference to the method flow chart and / or structural diagram provided by the embodiments of the present disclosure. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable application display device to generate a machine, so that the instructions executed by the processor of the computer or other programmable application display device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable application display device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable application display device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0121] The above disclosure is only a preferred embodiment of the present disclosure, which certainly cannot be used to limit the scope of rights of the present disclosure. Therefore, equivalent changes made according to the claims of the present disclosure are still within the scope of the present disclosure.

Claims

1. An image processing method, characterized in that: include: Acquire a first foreground image and a first background image; determining a first focus of the first foreground image and a second focus of the first background image; Determine first change line trajectory information of the first focus by sampling within a first motion speed range and a first motion angular velocity range; Determine second change line trajectory information of the second focus by sampling within a second motion speed range and a second motion angular velocity range; updating first position information of a first pixel in the first foreground image at each time according to the first change line trajectory information; updating second position information of a second pixel in the first background image at each time according to the second change line trajectory information; The first foreground image and the first background image at the same time after updating are combined to obtain a second image.

2. The method according to claim 1, characterized in that Determining first change line trajectory information of the first focus by sampling within a first motion speed range and a first motion angular speed range includes: determining a predetermined number of first time intervals; In the first time interval, sampling the first motion speed range and the first motion angular speed range to determine a first sampling lens motion trajectory of the first focus in the first time interval; Mapping the first sampled lens motion trajectory to an image space to obtain a first lens motion trajectory in the image space; In the first time interval, sampling the first starting point and the first end point of the first lens motion trajectory at intervals of angular velocity changes to obtain first sub-change line trajectory information of the first focus in the first time interval; The predetermined number of first sub-change line trajectory information are spliced ​​to obtain the first change line trajectory information.

3. The method according to claim 2, characterized in that When sampling is performed within the first motion speed range and the first motion angular speed range, the first motion angular speed is fixed within the first time interval; or the first motion angular speed is dynamically changed.

4. The method according to claim 2, characterized in that Determining second change line trajectory information of the second focus by sampling within a second motion speed range and a second motion angular velocity range includes: In the first time interval, sampling the second motion speed range and the second motion angular speed range to determine a second sampling lens motion trajectory of the second focus in the first time interval; Mapping the second sampling lens motion trajectory to the image space to obtain a second lens motion trajectory in the image space; In the first time interval, sampling the second starting point and the second end point of the second lens motion trajectory at intervals of angular velocity changes to obtain second sub-change line trajectory information of the second focus in the first time interval; The predetermined number of second sub-change line trajectory information are spliced ​​to obtain the second change line trajectory information.

5. The method according to claim 1, characterized in that The first range of motion speed is different from the second range of motion speed; and / or, The first movement angular velocity range is different from the second movement angular velocity range.

6. The method according to claim 1, characterized in that Obtaining a first foreground image and a first background image, including: Get the first video; Extracting the object contained in the first video by using a semantic segmentation model to obtain first background data and first foreground data containing the object; determining the first foreground image including the object from the first foreground data; The first background data is completed by generating a network model to obtain the first background image.

7. The method according to claim 1, characterized in that Combining the first foreground image and the first background image at the same time after updating to obtain a second image, comprising: The first foreground image and the different first background images at the same time after updating, which contain different objects, are respectively combined to obtain the second image.

8. The method according to claim 1, characterized in that Also includes: The plurality of the second images are stitched together to obtain a second video.

9. The method according to claim 1, characterized in that Also includes: The deep learning model is trained using the second image.

10. An image processing device, characterized in that: include: An acquisition unit, used for acquiring a first foreground image and a first background image; a determining unit, configured to determine a first focus of the first foreground image and a second focus of the first background image; A sampling unit, configured to determine first change line trajectory information of the first focus by sampling within a first motion speed range and a first motion angular velocity range; The sampling unit is further used to determine the second change line trajectory information of the second focus by sampling within a second movement speed range and a second movement angular velocity range; an updating unit, configured to update first position information of a first pixel in the first foreground image at each moment according to the first changed line trajectory information; The updating unit is further used to update the second position information of the second pixel in the first background image at each time according to the second change line trajectory information; A combining unit is used to combine the first foreground image and the first background image at the same time after updating to obtain a second image.

11. A computer device, characterized in that: Includes processor, memory, input and output interfaces; The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images