Method for training prediction network, image processing method and device

Abstract

The present disclosure provides a method for training a prediction network, an image processing method, an apparatus, a device, and a storage medium, relating to the field of artificial intelligence, specifically computer vision and deep learning technologies. A specific implementation scheme comprises: for each of a plurality of sample sets, inputting the sample set into a prediction network, wherein each of the plurality of sample sets includes background histogram samples and corresponding foreground histogram samples; determining a loss value based on a standard histogram corresponding to the sample set and a prediction result; and, if the loss value is greater than a loss threshold, adjusting parameters of the prediction network based on the loss value.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to computer vision and deep learning technologies. Background Art

[0002] With the rise of remote work in recent years, video conferencing software has become increasingly popular. A common feature in video conferencing software is background replacement, which removes the foreground image and replaces it with the background. This helps protect privacy and improve meeting quality. Summary of the Invention

[0003] The present disclosure provides a method for training a prediction network, an image processing method, an apparatus, a device, and a storage medium.

[0004] According to one aspect of the present disclosure, a method for training a prediction network is provided, comprising: for each sample set in a plurality of sample sets, inputting the sample set into the prediction network, wherein each sample set in the plurality of sample sets includes background histogram samples and corresponding foreground histogram samples; determining a loss value based on a standard histogram corresponding to the sample set and the prediction result; and, when the loss value is greater than a loss threshold, adjusting parameters of the prediction network according to the loss value.

[0005] According to another aspect of the present disclosure, an image processing method is provided, comprising: determining a background histogram of a background image and a first foreground histogram of a first foreground image; inputting the background histogram and the first foreground histogram into a prediction network to obtain a prediction histogram; determining a first target foreground image based on the prediction histogram and the first foreground histogram; and synthesizing the background image with the first target foreground image to obtain a first target image, wherein the prediction network is trained according to the method for training a prediction network described in an embodiment of the present disclosure.

[0006] According to another aspect of the present disclosure, a device for training a prediction network is provided, comprising: an input module for inputting, for each sample set in a plurality of sample sets, the sample set into the prediction network, wherein each sample set in the plurality of sample sets includes background histogram samples and corresponding foreground histogram samples; a loss determination module for determining a loss value based on a standard histogram corresponding to the sample set and the prediction result; and an adjustment module for adjusting parameters of the prediction network according to the loss value when the loss value is greater than a loss threshold.

[0007] According to another aspect of the present disclosure, an image processing device is provided, comprising: a first determination module for determining a background histogram of a background image and a first foreground histogram of a first foreground image; an input module for inputting the background histogram and the first foreground histogram into a prediction network to obtain a prediction histogram; a second determination module for determining a first target foreground image based on the prediction histogram and the first foreground histogram; and a synthesis module for synthesizing the background image with the first target foreground image to obtain a first target image, wherein the prediction network is trained according to the method for training a prediction network described in an embodiment of the present disclosure.

[0008] Another aspect of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method shown in the embodiment of the present disclosure.

[0009] According to another aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method shown in the embodiment of the present disclosure.

[0010] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction, wherein the computer program / instruction implements the steps of the method shown in the embodiment of the present disclosure when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1A Schematically shows a background image diagram according to an embodiment of the present disclosure;

[0014] Figure 1B Schematically shows a schematic diagram of a video image according to an embodiment of the present disclosure;

[0015] Figure 1C Schematically shows a schematic diagram of a composite image according to an embodiment of the present disclosure;

[0016] Figure 2 Schematically shows a flow chart of a method for training a prediction network according to an embodiment of the present disclosure;

[0017] Figure 3 Schematically shows a prediction network diagram according to an embodiment of the present disclosure;

[0018] Figure 4 A schematic diagram schematically illustrates a method for training a prediction network according to an embodiment of the present disclosure;

[0019] Figure 5 The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown;

[0020] Figure 6 The following schematically shows a flow chart of an image processing method according to another embodiment of the present disclosure;

[0021] Figure 7 A block diagram schematically illustrates an apparatus for training a prediction network according to an embodiment of the present disclosure;

[0022] Figure 8 A block diagram schematically shows an image processing apparatus according to an embodiment of the present disclosure; and

[0023] Figure 9 A block diagram schematically illustrates an example electronic device that may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] The following will be combined Figures 1A to 1C The application scenarios of the method and device provided by the present disclosure are described.

[0026] It should be noted that the following are only examples of application scenarios in which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0027] The image processing method according to the embodiment of the present disclosure can be used, for example, to replace the background portion in an image.

[0028] Based on this, Figure 1A Schematically shows a background image diagram according to an embodiment of the present disclosure. Figure 1A As shown, for example, the background image 110 shows an office environment.

[0029] Figure 1B Schematically shows a video image diagram according to an embodiment of the present disclosure. Figure 1B As shown, the video image schematic diagram 120 includes a human body image 121 .

[0030] According to an embodiment of the present disclosure, for reasons such as protecting privacy or achieving better video quality, the background of the human image 121 in the video image 120 can be replaced. Based on this, the human image 121 can be extracted from the video image 120. The human image 121 is then used as the foreground image and synthesized with the background image 110 to obtain a synthesized image 130.

[0031] Figure 1C Schematically shows a schematic diagram of a composite image according to an embodiment of the present disclosure. Figure 1C As shown, the background portion of the original video image 120 is replaced with the background image 110 , so that the human body image 121 and the background image 110 are synthesized into a synthesized image 130 , achieving the effect of placing the human body in an office environment.

[0032] However, when the shooting environments of the background image 110 and the video image 120 are different, in the composite image 130 after the background is replaced, the human image 121 and the background image 110 have large differences in color and lighting, resulting in a strong sense of incongruity in the composite image 130.

[0033] Based on this, according to an embodiment of the present disclosure, a prediction network can be pre-trained. The input of the prediction network is the histogram of the background image and the histogram of the foreground image, and the output is another histogram. The histogram contains the color and lighting information after the background image and the foreground image are synthesized. On this basis, the background histogram of the background image 120 and the foreground histogram of the human body image 121 as the foreground image can be determined. The background histogram and the foreground histogram are then input into the pre-trained prediction network to obtain a predicted histogram. Next, the target foreground image is determined based on the predicted histogram and the foreground histogram. The background image 120 is synthesized with the target foreground image to obtain a synthesized image 130, so that the color and lighting of the foreground image 121 in the synthesized image 130 can be made to better match the background image 110, reducing the sense of incongruity.

[0034] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0035] Figure 2 The flowchart of the method for training a prediction network according to an embodiment of the present disclosure is schematically shown.

[0036] like Figure 2 As shown, the method 200 for training a prediction network includes, in operation S210 , inputting each sample set in a plurality of sample sets into the prediction network.

[0037] According to an embodiment of the present disclosure, each sample set in the plurality of sample sets includes background histogram samples and corresponding foreground histogram samples.

[0038] Then, in operation S220, a loss value is determined based on the standard histogram corresponding to the sample set and the prediction result.

[0039] In operation S230 , when the loss value is greater than the loss threshold, parameters of the prediction network are adjusted according to the loss value.

[0040] According to an embodiment of the present disclosure, background histogram samples and corresponding foreground histogram samples in a sample set can be generated based on a standard image. The standard image includes a background portion and a foreground portion. According to an embodiment of the present disclosure, a foreground image and a background image in the standard image can be extracted. The foreground image can then be adjusted so that the foreground image does not match the background image. The adjustment can be performed at least once, thereby obtaining at least one adjusted foreground image, i.e., a target foreground image. Next, a histogram of at least one target foreground image can be determined as a foreground histogram sample, a histogram of a background image can be determined as a background histogram sample, and a histogram of a standard image can be determined as a standard histogram.

[0041] In this embodiment, for example, foreground histogram samples and background histogram samples corresponding to the same standard image can be used as a sample set. It is understandable that when training the prediction network, the sample set can use the standard histogram of the standard image as the expected output.

[0042] Exemplarily, in this embodiment, images captured in real scenes may be collected as standard images.

[0043] Reference below Figure 3 , the prediction network shown above is further explained in conjunction with specific embodiments.

[0044] Figure 3 The figure schematically shows a prediction network diagram according to an embodiment of the present disclosure.

[0045] like Figure 3 As shown, the prediction network 310 may include multiple one-dimensional convolutional layers. Exemplarily, in this embodiment, the prediction network 310 may include, for example, 8 one-dimensional convolutional layers.

[0046] According to an embodiment of the present disclosure, the input of the prediction network 310 can be a background histogram 301 and a foreground histogram 302. The background histogram 301 and the foreground histogram 302 can each include a histogram of three channels, namely a red channel, a green channel, and a blue channel. The size of the background histogram 301 and the foreground histogram 302 can be 3*256, where 3 represents three channels and 256 represents that the pixel brightness range (0, 255) contains 256 brightness values.

[0047] According to an embodiment of the present disclosure, the output of the prediction network 310 may be a histogram 303. The histogram 303 also includes histograms of the same three channels as the background histogram 301 and the foreground histogram 302, and the size is also 3*256.

[0048] Reference below Figure 4 , the method for training the prediction network shown above is further explained in conjunction with specific embodiments.

[0049] Figure 4 A schematic diagram schematically illustrates a method for training a prediction network according to an embodiment of the present disclosure.

[0050] like Figure 4 As shown, a sample set can be obtained from multiple sample sets, and the sample set includes background histogram samples Fa and foreground histogram samples Fb. The expected output corresponding to the sample set is a standard histogram Fc.

[0051] According to an embodiment of the present disclosure, background histogram samples Fa and corresponding foreground histogram samples Fb can be input into a prediction network to obtain a prediction result Fd. A loss function is then used to calculate the loss between the prediction result Fd and the standard histogram Fc. The loss value can be used to represent the difference between the prediction result Fd and the expected output. In this embodiment, the expected output is the standard histogram Fc corresponding to the input sample set.

[0052] Next, a determination is made as to whether the loss value is greater than a loss threshold. If so, the parameters of the prediction network are adjusted, and another sample set is selected to continue the training process. If the loss value is less than or equal to the loss threshold, training is terminated. According to embodiments of the present disclosure, the loss threshold can be determined based on actual needs, and this disclosure does not impose any specific limitations on the specific value of the loss threshold.

[0053] For example, in this embodiment, the loss value can be calculated according to the following formula:

[0054]

[0055] Where L is the loss value, n is the total number of brightness values ​​in the prediction result, yi is the number of pixels corresponding to the i-th brightness value in the standard histogram, and xi is the number of pixels corresponding to the i-th brightness value in the prediction result.

[0056] According to an embodiment of the present disclosure, a histogram output by network prediction can be used to perform relighting processing on a foreground image, so that the color and lighting in the foreground image are more consistent with the background.

[0057] Based on this, Figure 5 The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown.

[0058] like Figure 5 As shown, the image processing method 500 includes determining a background histogram of a background image and a foreground histogram of a foreground image in operation S510.

[0059] In operation S520, the background histogram and the foreground histogram are input into a prediction network to obtain a prediction histogram.

[0060] According to an embodiment of the present disclosure, the prediction network is trained according to the method for training a prediction network according to an embodiment of the present disclosure.

[0061] In operation S530, a target foreground image is determined based on the prediction histogram and the foreground histogram.

[0062] According to an embodiment of the present disclosure, operation S530 may include, for example, performing histogram equalization on the prediction histogram and the foreground histogram, and then performing migration on the foreground histogram using the equalized prediction histogram as a target to obtain a target foreground image.

[0063] In operation S540, the background image is synthesized with the target foreground image to obtain a target image.

[0064] According to the embodiments of the present disclosure, by using the histogram output by the prediction network to relight the foreground image, the color and lighting in the foreground image can be made more consistent with the background. As a result, after the foreground image and the background image are synthesized, the synthesis effect is more realistic and natural.

[0065] The image processing method according to the embodiment of the present disclosure can also be used to process frame images in a video stream. Based on this, Figure 6 The flowchart of an image processing method according to another embodiment of the present disclosure is schematically shown.

[0066] like Figure 6 As shown, the image processing method 600 includes obtaining an initial frame image in a video stream in operation S610.

[0067] According to an embodiment of the present disclosure, the initial frame image may be, for example, a frame image corresponding to the first frame in a video stream.

[0068] In operation S620, an image containing the target object in the initial frame image is extracted as a first foreground image.

[0069] According to an embodiment of the present disclosure, the target object may include, for example, a human body. According to other embodiments of the present disclosure, the target object may also include other objects other than a human body that need to be placed in the foreground.

[0070] In operation S630, a background histogram of the background image and a first foreground histogram of the first foreground image are determined.

[0071] In operation S640, the background histogram and the first foreground histogram are input into a prediction network to obtain a prediction histogram.

[0072] In operation S650, a first target foreground image is determined based on the prediction histogram and the first foreground histogram.

[0073] According to an embodiment of the present disclosure, for example, histogram equalization processing can be performed on the prediction histogram and the first foreground histogram. Then, using the equalized prediction histogram as a target, the first foreground histogram is subjected to migration processing to obtain a first target foreground image.

[0074] In operation S660, the background image is synthesized with the first target foreground image to obtain a first target image.

[0075] In operation S670, other frame images except the initial frame image in the video stream are acquired, and then operations S680 to S6110 may be performed for each of the other frame images.

[0076] In operation S680, an image including the target object is extracted from the frame image as a second foreground image.

[0077] In operation S690, a second foreground histogram of the second foreground image is determined.

[0078] In operation S6100, a second target foreground image is determined based on the prediction histogram and the second foreground histogram.

[0079] According to an embodiment of the present disclosure, the method for determining the second target foreground image may refer to the above method for determining the first target foreground image, and will not be described in detail here.

[0080] In operation S6110, the background image is synthesized with the second foreground image to obtain a second target image.

[0081] According to the embodiments of the present disclosure, except for the first frame that needs to be predicted using the prediction network, the subsequent frames can directly perform migration processing using the previously obtained prediction histogram, thereby achieving faster processing speed. In addition, in the related art, the method of relighting the image is directly applied to the video stream, which often causes problems such as jitter in the result. However, when processing the frame images of the video stream using the image processing method according to the embodiments of the present disclosure, since all frame images are migrated using the same prediction histogram as the target, the timing is relatively stable, so there will be no jitter.

[0082] The following further describes the apparatus for training a prediction network shown in the embodiment of the present disclosure.

[0083] Figure 7 A block diagram schematically shows an apparatus for training a prediction network according to an embodiment of the present disclosure.

[0084] like Figure 7 As shown, the apparatus 700 for training a prediction network includes an input module 710 , a loss determination module 720 , and an adjustment module 730 .

[0085] The input module 710 is configured to input each sample set in a plurality of sample sets into a prediction network, wherein each sample set in the plurality of sample sets includes background histogram samples and corresponding foreground histogram samples.

[0086] The loss determination module 720 is configured to determine a loss value based on the prediction result and a standard histogram corresponding to the sample set.

[0087] The adjustment module 730 is used to adjust the parameters of the prediction network according to the loss value when the loss value is greater than the loss threshold.

[0088] According to an embodiment of the present disclosure, the loss determination module may include a calculation submodule for calculating the loss value according to the following formula:

[0089]

[0090] Wherein, L is the loss value, n is the total number of brightness values ​​in the prediction result, yi is the number of pixels corresponding to the i-th brightness value in the standard histogram, and xi is the number of pixels corresponding to the i-th brightness value in the prediction result.

[0091] According to an embodiment of the present disclosure, the above-mentioned device may further include an image extraction module, an adjustment module, a first histogram determination module, a second histogram determination module, and a third histogram determination module. The image extraction module is used to extract the foreground image and the background image in each standard image of at least one standard image. The adjustment module is used to adjust the foreground image to obtain at least one target foreground image. The first histogram determination module is used to determine the histogram of the at least one target foreground image as the foreground histogram sample. The second histogram determination module is used to determine the histogram of the background image as the background histogram sample. The third histogram determination module is used to determine the histogram of the standard image as the standard histogram.

[0092] According to an embodiment of the present disclosure, the prediction network may include, for example, multiple one-dimensional convolutional layers.

[0093] The image processing device shown in the embodiment of the present disclosure is further described below.

[0094] Figure 8 The following schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0095] like Figure 8 As shown, the image processing apparatus 800 includes a first determination module 810 , an input module 820 , a second determination module 830 and a synthesis module 840 .

[0096] The first determining module 810 is configured to determine a background histogram of the background image and a first foreground histogram of the first foreground image.

[0097] The input module 820 is used to input the background histogram and the first foreground histogram into the prediction network to obtain a predicted histogram.

[0098] The second determining module 830 is configured to determine a first target foreground image according to the predicted histogram and the first foreground histogram.

[0099] The synthesis module 840 is configured to synthesize the background image and the first target foreground image to obtain the first target image. The prediction network is trained according to the method for training a prediction network according to an embodiment of the present disclosure.

[0100] According to an embodiment of the present disclosure, the second determination module may include an equalization processing submodule and a migration processing submodule. The equalization processing submodule is configured to perform histogram equalization processing on the prediction histogram and the first foreground histogram. The migration processing submodule is configured to perform migration processing on the first foreground histogram using the equalized prediction histogram as a target to obtain a first target foreground image.

[0101] According to an embodiment of the present disclosure, the apparatus may further include a first acquisition module and a first extraction module. The first acquisition module is configured to acquire an initial frame image from the video stream. The first extraction module is configured to extract an image containing the target object from the initial frame image as the first foreground image.

[0102] According to an embodiment of the present disclosure, the above-mentioned device may further include a second acquisition module, a second extraction module, a third determination module, a fourth determination module and a second synthesis module. The second acquisition module is used to acquire other frame images in the video stream except the initial frame image. The second extraction module is used to extract, for each frame image in the other frame images, an image containing a target object in the frame image as a second foreground image. The third determination module is used to determine a second foreground histogram of the second foreground image. The fourth determination module is used to determine a second target foreground image based on the predicted histogram and the second foreground histogram. The second synthesis module is used to synthesize the background image with the second target foreground image to obtain a second target image.

[0103] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0104] Figure 9 A block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0105] like Figure 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0106] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0107] The computing unit 901 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the method for training the prediction network and the image processing method. For example, in some embodiments, the method for training the prediction network and the image processing method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the method for training the prediction network and the image processing method described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the method for training a prediction network and the image processing method in any other appropriate manner (for example, by means of firmware).

[0108] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0110] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0113] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0114] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0115] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for training a prediction network, comprising: For each sample set in a plurality of sample sets, inputting the sample set into a prediction network to obtain a prediction result, wherein each sample set in the plurality of sample sets includes a background histogram sample and a corresponding foreground histogram sample; The loss value is calculated according to the following formula: Wherein, L is the loss value, n is the total number of brightness values ​​in the prediction result, and y is i is the number of pixels corresponding to the ith brightness value in the standard histogram corresponding to the sample set, and the x i is the number of pixels corresponding to the ith brightness value in the prediction result; and When the loss value is greater than a loss threshold, the parameters of the prediction network are adjusted according to the loss value.

2. The method according to claim 1, further comprising: For each standard image in the at least one standard image, extracting a foreground image and a background image from the standard image; Adjusting the foreground image to obtain at least one target foreground image; determining a histogram of the at least one target foreground image as the foreground histogram sample; Determining a histogram of the background image as the background histogram sample; as well as A histogram of the standard image is determined as the standard histogram.

3. The method according to claim 1 or 2, wherein: The prediction network includes multiple one-dimensional convolutional layers.

4. An image processing method, comprising: determining a background histogram of the background image and a first foreground histogram of the first foreground image; Inputting the background histogram and the first foreground histogram into a prediction network to obtain a prediction histogram; determining a first target foreground image according to the predicted histogram and the first foreground histogram; as well as synthesizing the background image and the first target foreground image to obtain a first target image, The prediction network is trained according to the method according to any one of claims 1-3.

5. The method according to claim 4, wherein Determining a first target foreground image according to the predicted histogram and the first foreground histogram includes: performing histogram equalization processing on the prediction histogram and the first foreground histogram; and The equalized predicted histogram is used as a target, and the first foreground histogram is subjected to migration processing to obtain a first target foreground image.

6. The method according to claim 4 or 5, further comprising: Get the initial frame image in the video stream; as well as An image containing a target object in the initial frame image is extracted as the first foreground image.

7. The method according to claim 6, further comprising: Acquire other frame images in the video stream except the initial frame image; For each frame image in the other frame images, extracting an image containing a target object from the frame image as a second foreground image; determining a second foreground histogram of the second foreground image; determining a second target foreground image according to the predicted histogram and the second foreground histogram; as well as The background image is synthesized with the second target foreground image to obtain a second target image.

8. A device for training a prediction network, comprising: An input module, configured to input each sample set from a plurality of sample sets into a prediction network to obtain a prediction result, wherein each sample set from the plurality of sample sets includes a background histogram sample and a corresponding foreground histogram sample; The calculation submodule is used to calculate the loss value according to the following formula: Wherein, L is the loss value, n is the total number of brightness values ​​in the prediction result, and y is i is the number of pixels corresponding to the ith brightness value in the standard histogram corresponding to the sample set, and the x i is the number of pixels corresponding to the ith brightness value in the prediction result; and An adjustment module is used to adjust the parameters of the prediction network according to the loss value when the loss value is greater than the loss threshold.

9. The apparatus according to claim 8, further comprising: an image extraction module, configured to extract, for each standard image in at least one standard image, a foreground image and a background image in the standard image; an adjustment module, configured to adjust the foreground image to obtain at least one target foreground image; a first histogram determination module, configured to determine a histogram of the at least one target foreground image as the foreground histogram sample; a second histogram determination module, configured to determine a histogram of the background image as the background histogram sample; as well as The third histogram determination module is configured to determine a histogram of the standard image as the standard histogram.

10. The device according to claim 8 or 9, wherein The prediction network includes multiple one-dimensional convolutional layers.

11. An image processing device, comprising: A first determining module, configured to determine a background histogram of the background image and a first foreground histogram of the first foreground image; An input module, configured to input the background histogram and the first foreground histogram into a prediction network to obtain a prediction histogram; a second determining module, configured to determine a first target foreground image based on the predicted histogram and the first foreground histogram; as well as A first synthesis module is configured to synthesize the background image and the first target foreground image to obtain a first target image. The prediction network is trained according to the method according to any one of claims 1-3.

12. The device according to claim 11, wherein The second determining module includes: an equalization processing submodule, configured to perform histogram equalization processing on the prediction histogram and the first foreground histogram; and The migration processing submodule is used to perform migration processing on the first foreground histogram with the equalized prediction histogram as a target to obtain a first target foreground image.

13. The apparatus according to claim 11 or 12, further comprising: A first acquisition module is used to acquire an initial frame image in a video stream; as well as The first extraction module is configured to extract an image containing a target object from the initial frame image as the first foreground image.

14. The apparatus according to claim 13, further comprising: A second acquisition module is used to acquire other frame images in the video stream except the initial frame image; A second extraction module is configured to extract, for each of the other frame images, an image containing a target object in the frame image as a second foreground image; a third determining module, configured to determine a second foreground histogram of the second foreground image; a fourth determining module, configured to determine a second target foreground image based on the predicted histogram and the second foreground histogram; as well as The second synthesis module is used to synthesize the background image and the second target foreground image to obtain a second target image.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images