Image processing method, device, equipment, medium and product

Abstract

The present invention discloses an image processing method, device, equipment, medium and product, which relate to the field of artificial intelligence technology, and in particular to the field of computer vision and deep learning technology. The specific implementation scheme is: obtaining a first image and a second image, wherein the first image and the second image include different target objects; extracting first object features in the first image, and extracting second object features in the second image, wherein the first object features and the second object features are object features of the target object in different dimensions; performing image synthesis based on the first object features and the second object features, and obtaining a third image including a new target object; the new target object has the first object features and the second object features. The present invention can realize the exchange of object features of different dimensions between different target objects, and synthesize the image of a new target object based on the exchanged object features, thereby obtaining more and richer object data samples.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of computer vision and deep learning technology. Background Art

[0002] With the advancement of science and technology, various applications of computer vision tasks play a vital role in people's work and life, and have a wide range of application scenarios. Data enhancement, that is, expanding training data through technical means to improve the generalization performance of the model, is of great significance to improving the performance of computer vision tasks.

[0003] In current technology, data enhancement can be achieved through general target detection data enhancement, for example, affine transformation of the original data, which may include color space transformation, scale transformation, rotation, mosaic data enhancement, random cropping, etc. Generative adversarial networks can also be used to generate random transformation data. Summary of the invention

[0004] The present disclosure provides a method, apparatus, device, medium and product for image processing.

[0005] According to one aspect of the present disclosure, an image processing method is provided, comprising: acquiring a first image and a second image, wherein the first image and the second image include different target objects; extracting first object features from the first image, and extracting second object features from the second image, wherein the first object features and the second object features are object features of the target object in different dimensions; performing image synthesis based on the first object features and the second object features to obtain a third image including a new target object; the new target object has the first object features and the second object features.

[0006] According to another aspect of the present disclosure, there is provided an image processing device, including: an acquisition module, used to acquire a first image and a second image, wherein the first image and the second image include different target objects; a determination module, used to extract first object features in the first image and second object features in the second image, wherein the first object features and the second object features are object features of the target object in different dimensions; and a synthesis module, used to perform image synthesis based on the first object features and the second object features to obtain a third image including a new target object, wherein the new target object has the first object features and the second object features.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image processing method described in any one of the present disclosure.

[0008] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the image processing methods of the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the image processing methods of the present disclosure.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is a flowchart of an image processing method according to the present disclosure;

[0013] Figure 2 is a schematic diagram of the training process of the adversarial network according to the present disclosure;

[0014] Figure 3 is a flowchart of an image processing method according to the present disclosure;

[0015] Figure 4 is a flowchart of a method for determining a first object feature reconstruction loss according to the present disclosure;

[0016] Figure 5 is a flow chart of a method for determining a second object feature reconstruction loss according to the present disclosure;

[0017] Figure 6 is a flow chart of a method for determining image reconstruction loss according to the present disclosure;

[0018] Figure 7 It is a flowchart of a method for synthesizing a first object feature and a second object feature to obtain a third image based on a first object feature reconstruction loss, a second object feature reconstruction loss and an image reconstruction loss according to the present disclosure;

[0019] Figure 8 is a schematic diagram of a process of applying the image processing method disclosed in the present invention to pedestrian detection;

[0020] Fig. 9 is a schematic diagram of applying the image processing method disclosed in the present invention to pedestrian recognition;

[0021] Fig.10 is a schematic diagram of realizing feature exchange of a target object based on a generative adversarial network according to the present disclosure;

[0022] Fig.11 is a block diagram of an image processing apparatus according to the present disclosure;

[0023] Fig.12 It is a block diagram of an electronic device used to implement the image processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] Smart construction sites are to organize and improve engineering construction and personnel arrangements with artificial intelligence methods, implanting artificial intelligence technologies such as sensor technology and virtual reality into various objects such as buildings, machinery, personnel wearable facilities, and site access points, and are generally interconnected to form an "Internet of Things", and then integrated with the "Internet" to achieve the integration of engineering management stakeholders and engineering construction sites. The core of smart construction sites is to improve the way in which various engineering stakeholders and post personnel interact with each other in a "smarter" way, so as to improve the clarity, efficiency, flexibility and response speed of interaction. The application of smart construction sites is to ensure the safety of personnel at work and facilitate the arrangement and management of personnel. Relying on video surveillance systems and using computer vision methods, pedestrian detection, identification, tracking and behavior analysis can be achieved, which is of great significance to the realization of smart construction sites. In computer vision models, the larger the amount of data, the better the model can learn the laws of the data, and the better the effect. Data enhancement technology, that is, through a series of technical means, the input data is transformed, the training data is expanded, and the generalization performance of the model is improved.

[0026] In current technology, in order to expand the data set, new images are generated by adding noise to the image, changing the image angle, mosaic data enhancement, changing the color space, etc. Simply processing the image will cause the background to be too different, affecting the training effect. In addition, using a generative adversarial network to randomly generate data, or performing cross-domain data transformation to generate new data, however, the quality of the generated images is not high, and using low-quality generated images for model training is not conducive to model training.

[0027] In view of this, an embodiment of the present disclosure provides an image processing method, which extracts object features of different dimensions from different target objects in different images, and combines features from different target objects to achieve the exchange of object features between different target objects, thereby obtaining a composite image.

[0028] Figure 1 is a flow chart of the image processing method according to the present disclosure. Figure 1 As shown, the method of the present disclosure includes the following steps.

[0029] In step S101 , a first image and a second image are acquired, wherein the first image and the second image include different target objects.

[0030] In the disclosed embodiment, a first image and a second image including a target object are acquired, and different target objects are included in the first image and the second image. It can be understood that different target objects have the same attributes, for example, they are pedestrians, animals, plants, or objects. Taking the target object as a pedestrian as an example, the first image and the second image are different images, and different pedestrians are included in the first image and the second image. For a construction site scene, the first image and the second image include different construction site pedestrians.

[0031] In step S102, a first object feature is extracted from the first image, and a second object feature is extracted from the second image, where the first object feature and the second object feature are object features of the target object in different dimensions.

[0032] Extract the first object feature of the target object in the first image, and extract the second object feature of the target object in the second image, the first object feature and the second object feature are the object features of the target object in different dimensions, so as to perform feature segmentation on the target object in the first image and the second image, and then combine the object features of different dimensions obtained by the segmentation. Still taking the target object as a pedestrian as an example, the first object feature can be the morphological feature of the pedestrian, such as posture, posture, body shape, etc. The second object feature can be the appearance feature of the pedestrian, such as appearance, clothing, etc.

[0033] In step S103, an image is synthesized based on the first object feature and the second object feature to obtain a third image including a new target object, where the new target object has the first object feature and the second object feature.

[0034] For the first object features of the target object in the first image and the second object features of the target object in the second image extracted, the first object features and the second object features belonging to different dimensions are combined to perform image synthesis to obtain a third image, which includes a new target object, and the new target object has the first object features of the target object in the first image and the second object features of the target object in the second image.

[0035] According to the embodiments of the present disclosure, object features of different dimensions of different target objects in different images are extracted, the features of the target objects are split, and features from different target objects are combined to achieve exchange of object features between different target objects and obtain a composite image, thereby generating more and richer data samples of the target objects.

[0036] In an exemplary embodiment of the present disclosure, the first object feature is extracted from the first image, the second object feature is extracted from the second image, and the image is synthesized based on the first object feature and the second object feature to obtain a third image including a new target object. The process can be implemented based on a generative adversarial network. The generative adversarial (GAN) network can realize feature exchange of the input image, and the backbone network can be replaced by a mainstream convolutional neural network, such as VGGNet, HRNet, Efficientnet, Moblenet or Transformer. For the generator and encoder implementation of the generative adversarial network used in the present invention, structures such as WGAN, SRGAN or VAEGAN can be used, and the embodiments of the present disclosure are not limited to this.

[0037] The generative adversarial network in the disclosed embodiment includes an encoder, a generator, and a discriminator. The encoder may be multiple, each used to extract different object features of the input image. The generator is used to synthesize images based on the object features to obtain an image including a new target object. By generating an adversarial network, a new image is obtained by exchanging object features for synthesis, which can achieve image data enhancement. The discriminator of the generative adversarial network judges the authenticity of the image reconstructed by the generator to better train the generator in the generative adversarial network.

[0038] In one embodiment of the present disclosure, the first object feature is a morphological feature of a pedestrian, and the second object feature is an appearance feature of the pedestrian. Accordingly, the encoder may include a morphological feature encoder and an appearance feature encoder. The appearance feature encoder extracts the appearance features of the pedestrian, including the appearance features of the pedestrian such as clothes, pants, and helmets, and the morphological feature encoder extracts the morphological information of the pedestrian. The generator performs image synthesis based on the morphological features and the appearance features to reconstruct a new image.

[0039] It should be noted that the generative adversarial network in the embodiment of the present disclosure can be obtained through training, and the trained generative adversarial network is subsequently used to reconstruct images and perform data enhancement for application scenarios such as pedestrian detection and pedestrian recognition. During the generative adversarial network training process, the network loss value is determined using the re-identification loss function, and the parameters of each layer of the network are optimized to obtain a trained generative adversarial network that meets the conditions. For example, the gradient of the parameters of the network can be calculated by back propagation, and the parameters of each layer of the network can be optimized by an optimization algorithm.

[0040] In one embodiment of the present disclosure, the process of training a generative adversarial network includes preprocessing of training data samples, and the preprocessing includes labeling pedestrian data, labeling different pedestrian objects with different, unique IDs, and the pedestrian object corresponding to the same ID may include multiple images as data samples.

[0041] Figure 2 is a schematic diagram of the training process of the adversarial network according to the present disclosure, referring to Figure 2The input of the generative adversarial network used in the embodiment of the present disclosure is a triplet input, that is, each group of inputs of the generative adversarial network includes three pictures, namely, a first image and a second image. The first image and the second image include different target objects. For the second image, there is a first sub-image and a second sub-image. It can be understood that the first sub-image and the second sub-image have the same target object. The first image is input into the first object feature encoder to obtain the first object feature of the first image, and the first sub-image of the second image is input into the first object encoder and the second object encoder to obtain the first object feature of the first sub-image and the second object feature of the first sub-image respectively. The second sub-image of the second image is input into the second object encoder to obtain the second object feature of the second sub-image. The first object feature and the second object feature of the first sub-image are input into the generator for reconstruction to obtain the reconstructed image of the first sub-image. The first object feature of the first sub-image and the second object feature of the second sub-image are input into the generator for reconstruction to obtain the reconstructed image of the second sub-image. The first object feature of the first image and the second object feature of the first sub-image are input into the generator for reconstruction to obtain the reconstructed image of the first image. The first object feature of the first image and the second object feature of the first sub-image are input into the generator for reconstruction to obtain the reconstructed image of the first image. The reconstructed image of the first image is input into the first object encoder and the second object encoder to obtain the first object feature and the second object feature of the reconstructed image of the first image.

[0042] Figure 3 is a flow chart of the image processing method according to the present disclosure. Figure 3 As shown, the method of the present disclosure includes the following steps.

[0043] In step S201 , a first object feature reconstruction loss, a second object feature reconstruction loss, and an image reconstruction loss are determined.

[0044] In step S202, the first object feature and the second object feature are synthesized based on the first object feature reconstruction loss, the second object feature reconstruction loss and the image reconstruction loss to obtain a third image.

[0045] In the disclosed embodiment, the first object feature reconstruction loss is the reconstruction loss between the first object feature of the first image and the first object feature of the reconstructed image of the first image. The second object feature reconstruction loss includes the reconstruction loss between the second object feature of the second image and the second object feature of the reconstructed image of the first image, and the reconstruction loss between the second object feature of the second image and the second object feature of the reconstructed image of the second image. The image reconstruction loss is the loss between the second image and the reconstructed image of the second image. Based on the first object feature reconstruction loss, the second object feature reconstruction loss, and the image reconstruction loss, the network is constrained during the training of the adversarial network to synthesize a third image including the first object feature and the second object feature.

[0046] According to the embodiments of the present disclosure, the first object feature and the second object feature are synthesized through the first object feature reconstruction loss, the second object feature reconstruction loss and the image reconstruction loss to obtain a third image. Compared with the unconstrained generative adversarial network that introduces random noise for data enhancement, it is possible to utilize object features of different dimensions from real samples to achieve the exchange of object features of different target objects, effectively improve the authenticity of the synthesized image, and reconstruct and synthesize more pedestrian samples under limited samples to achieve data enhancement, thereby utilizing the synthesized pedestrian data to effectively improve the accuracy in pedestrian detection and pedestrian re-identification applications.

[0047] Figure 4 FIG. 1 is a flow chart of a method for determining a first object feature reconstruction loss according to the present disclosure. Figure 4 As shown, the method of the present disclosure includes the following steps.

[0048] In step S301, the first object feature and the second object feature are synthesized to obtain a reconstructed image of the first image.

[0049] In step S302, a first object feature is extracted from a reconstructed image of the first image.

[0050] In step S303, a loss between the first object feature extracted in the first image and the first object feature extracted in the reconstructed image of the first image is determined.

[0051] Still refer to Figure 2 In the training process of the generative adversarial model in the present disclosure, in an exemplary embodiment, the first object feature of the first image and the second object feature of the first sub-image are input into the generator for reconstruction to obtain a reconstructed image of the first image, and the first object feature is extracted from the reconstructed image of the first image. The loss between the first object feature extracted from the first image and the first object feature extracted from the reconstructed image of the first image is determined. It can be understood that the first object feature extracted from the reconstructed image of the first image and the first object feature extracted from the first image belong to the same target object, that is, the first object encoder in the generative adversarial network can be trained based on the loss between the two. For example, it can be based on the feature distance between the first object feature extracted from the reconstructed image of the first image and the first object feature extracted from the first image. The smaller the feature distance, the better the learning effect of the trained first object encoder.

[0052] According to an embodiment of the present disclosure, effective training of the first object encoder of the generative adversarial network is achieved through the loss between the first object feature extracted in the first image and the first object feature extracted in the reconstructed image of the first image, which can improve the training effect of the generative adversarial network and thus improve the quality of the synthesized image.

[0053] Figure 5 FIG. 1 is a flow chart of a method for determining a second object feature reconstruction loss according to the present disclosure. Figure 5 As shown, the method of the present disclosure includes the following steps.

[0054] In step S401, a second object feature is extracted from a first sub-image, and a second object feature is extracted from a second sub-image.

[0055] In step S402, the first object feature extracted from the first image is synthesized with the second object feature extracted from the first sub-image to obtain a reconstructed image of the first image.

[0056] In step S403, a second object feature is extracted from the reconstructed image of the first image.

[0057] In step S404, the loss between the second object feature extracted in the first sub-image and the second object feature extracted in the second sub-image is determined, and the loss between the second object feature extracted in the first sub-image and the second object feature extracted in the reconstructed image of the first image is determined.

[0058] Still refer to Figure 2 In the training process of the generative adversarial model, in an exemplary embodiment of the present disclosure, the second image includes a first sub-image and a second sub-image, and the loss between the second object feature extracted from the first sub-image of the second image and the second object feature extracted from the second sub-image of the second image is determined, and the loss between the second object feature extracted from the first sub-image of the second image and the second object feature extracted from the reconstructed image of the first image is determined. It can be understood that the second object feature used to determine the loss of the second object feature belongs to the same target object. The second object encoder of the generative adversarial network is trained using the second object feature reconstruction loss to improve the training effect of the second object encoder.

[0059] According to an embodiment of the present disclosure, the loss between the second object feature extracted from the first subimage of the second image and the second object feature extracted from the second subimage of the second image is determined, and the loss between the second object feature extracted from the first subimage of the second image and the second object feature extracted from the reconstructed image of the first image is determined, so as to achieve effective training of the second object encoder of the generative adversarial network, improve the training effect of the generative adversarial network, and thus improve the quality of the synthesized image.

[0060] Figure 6 FIG. 1 is a flow chart of a method for determining image reconstruction loss according to the present disclosure. Figure 6 As shown, the method of the present disclosure includes the following steps.

[0061] In step S501, first object features and second object features are extracted from a first sub-image, and second object features are extracted from a second sub-image.

[0062] In step S502, the first object feature of the first sub-image and the second object feature of the first sub-image are synthesized to obtain a reconstructed image of the first sub-image.

[0063] In step S503, the first object feature of the first sub-image and the second object feature of the second sub-image are synthesized to obtain a reconstructed image of the second sub-image.

[0064] In step S504 , an image reconstruction loss between a first sub-image and a reconstructed image of the first sub-image is determined, and a reconstruction loss between a second sub-image and a reconstructed image of the second sub-image is determined.

[0065] Still refer to Figure 2 In the training process of the generative adversarial model in the exemplary embodiment of the present disclosure, the second image includes a first sub-image and a second sub-image, and the first sub-image and the second sub-image correspond to the same target object. The first object feature of the first sub-image and the second object feature of the first sub-image are synthesized to obtain a reconstructed image of the first sub-image. The first object feature of the first sub-image and the second object feature of the second sub-image are synthesized to obtain a reconstructed image of the second sub-image. The image reconstruction loss between the first sub-image and the reconstructed image of the first sub-image is determined, and the reconstruction loss between the second sub-image and the reconstructed image of the second sub-image is determined. The generator of the generative adversarial network is trained using the image reconstruction loss to improve the training effect of the second object encoder.

[0066] According to an embodiment of the present disclosure, features of a first sub-image and a second sub-image belonging to the same target object of a second image are exchanged, and a generative adversarial network is trained based on the image reconstruction loss between the reconstructed image and the original image. This can improve the training effect of the generative adversarial network and thus improve the quality of the synthesized image.

[0067] Figure 7 The present invention is a flowchart of a method for synthesizing a first object feature and a second object feature to obtain a third image based on a first object feature reconstruction loss, a second object feature reconstruction loss, and an image reconstruction loss. Figure 7 As shown, the method of the present disclosure includes the following steps.

[0068] In step S601 , a re-identification loss function is determined based on a first object feature reconstruction loss, a second object feature reconstruction loss, and an image reconstruction loss.

[0069] In step S602, based on a generative adversarial network constrained by a re-identification loss function, the first object feature and the second object feature are synthesized to obtain a third image.

[0070] In the disclosed embodiment, based on a generative adversarial network constrained by a re-identification loss function, the first object feature and the second object feature are synthesized to obtain a third image, which can improve the quality of the reconstructed synthesized image and the authenticity of the synthesized image, thereby obtaining more and richer pedestrian image data. The use of pedestrian image data can effectively improve the accuracy and recall rate of pedestrian detection and pedestrian re-identification.

[0071] Figure 8 is a flow chart of applying the image processing method disclosed in the present invention to pedestrian detection. Figure 8 In the process, a first image is obtained, the first image includes a target object to be detected as a pedestrian, and a second image is obtained, the second image can be obtained based on a target domain pedestrian dataset. Using the generative adversarial network of the embodiment of the present disclosure, the generative adversarial network includes a morphological feature encoder, an appearance feature encoder and a generator, the morphological features in the first image are extracted, and the appearance features in the second image are extracted, the morphological features of the first image and the appearance features of the second image are input into the generator, and a third image of the first image with the appearance features exchanged is reconstructed. The third image can be used as the added first image to achieve data enhancement of the target object to be detected as a pedestrian.

[0072] Fig. 9 This is a schematic diagram of applying the image processing method disclosed in the present invention to pedestrian recognition. M images of the first pedestrian are used as the first image, and N images of the second pedestrian are used as the second image. The features of the first pedestrian and the second pedestrian of different dimensions are exchanged through the generative adversarial network of the embodiment of the present invention, so that M×N pedestrian images can be obtained, thereby realizing pedestrian data enhancement.

[0073] Fig.10 It is a schematic diagram of realizing feature exchange of target objects based on a generative adversarial network according to the present disclosure. A first image and a second image are obtained, the first image and the second image include different target objects, and the image synthesis is performed using the generative adversarial network of the embodiment of the present disclosure. The generative adversarial network includes a morphological feature encoder, an appearance feature encoder, and a generator. The morphological feature encoder is used to extract morphological features in the first image, and the appearance feature encoder is used to extract appearance features in the second image. The morphological features of the first image and the appearance features of the second image are input into the generator, and the images are synthesized to obtain a third image. The target object included in the third image has the morphological features of the target object in the first image and the morphological features of the target object in the second image.

[0074] Based on the same concept, an embodiment of the present disclosure also provides an image processing device.

[0075] It is understandable that the device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0076] Fig.11 is a block diagram of an image processing apparatus according to the present disclosure.

[0077] like Fig.11 As shown, the image processing device 700 of the embodiment of the present disclosure includes: an acquisition module 701, a determination module 702 and a synthesis module 703.

[0078] The acquisition module 701 is used to acquire a first image and a second image, wherein the first image and the second image include different target objects.

[0079] The determination module 702 is used to extract a first object feature from the first image and a second object feature from the second image, wherein the first object feature and the second object feature are object features of the target object in different dimensions.

[0080] The synthesis module 703 is used to perform image synthesis based on the first object feature and the second object feature to obtain a third image including a new target object, where the new target object has the first object feature and the second object feature.

[0081] In an exemplary embodiment of the present disclosure, the synthesis module 703 is further used to: determine a first object feature reconstruction loss, a second object feature reconstruction loss and an image reconstruction loss, the first object feature reconstruction loss being the reconstruction loss between the first object feature of the first image and the first object feature of the reconstructed image of the first image; the second object feature reconstruction loss comprising the reconstruction loss between the second object feature of the second image and the second object feature of the reconstructed image of the first image, and the reconstruction loss between the second object feature of the second image and the second object feature of the reconstructed image of the second image, the image reconstruction loss being the loss between the second image and the reconstructed image of the second image; based on the first object feature reconstruction loss, the second object feature reconstruction loss and the image reconstruction loss, the first object feature and the second object feature are synthesized to obtain a third image.

[0082] In an exemplary embodiment of the present disclosure, the synthesis module 703 is also used to: synthesize the first object feature and the second object feature to obtain a reconstructed image of the first image; extract the first object feature from the reconstructed image of the first image; and determine the loss between the first object feature extracted in the first image and the first object feature extracted in the reconstructed image of the first image.

[0083] In an exemplary embodiment of the present disclosure, the second image includes a first sub-image and a second sub-image, and the first sub-image and the second sub-image include the same target object; the synthesis module 703 is also used to: extract the second object feature in the first sub-image, and extract the second object feature in the second sub-image; synthesize the first object feature extracted in the first image with the second object feature extracted in the first sub-image to obtain a reconstructed image of the first image; extract the second object feature in the reconstructed image of the first image; determine the loss between the second object feature extracted in the first sub-image and the second object feature extracted in the second sub-image, and determine the loss between the second object feature extracted in the first sub-image and the second object feature extracted in the reconstructed image of the first image.

[0084] In an exemplary embodiment of the present disclosure, the second image includes a first sub-image and a second sub-image, and the first sub-image and the second sub-image include the same target object; the synthesis module 703 is also used to: extract first object features and second object features in the first sub-image, and extract the second object features in the second sub-image; synthesize the first object features of the first sub-image and the second object features of the first sub-image to obtain a reconstructed image of the first sub-image; synthesize the first object features of the first sub-image and the second object features of the second sub-image to obtain a reconstructed image of the second sub-image; determine the image reconstruction loss between the first sub-image and the reconstructed image of the first sub-image, and determine the reconstruction loss between the second sub-image and the reconstructed image of the second sub-image.

[0085] In an exemplary embodiment of the present disclosure, the synthesis module 703 is further used to: determine a re-identification loss function based on a first object feature reconstruction loss, a second object feature reconstruction loss, and an image reconstruction loss; and synthesize the first object feature and the second object feature based on a generative adversarial network constrained by the re-identification loss function to obtain a third image.

[0086] In summary, according to the image processing device of the embodiment of the present disclosure, object features of different dimensions of different target objects in different images are extracted, the features of the target objects are split, and features from different target objects are combined to achieve the exchange of object features between different target objects to obtain a composite image, thereby generating more and richer data samples of the target objects.

[0087] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0088] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0089] Fig.12 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0090] like Fig.12 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0091] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0092] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the startup image processing method in any other appropriate manner (e.g., by means of firmware).

[0093] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0095] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0097] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0098] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0099] According to the technical solution provided by the embodiments of the present disclosure, object features of different dimensions of different target objects in different images are extracted, the features of the target objects are split, and the features from different target objects are combined to achieve the exchange of object features between different target objects and obtain a synthetic image, thereby generating more and richer data samples of the target objects.

[0100] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0101] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Acquire a first image and a second image, wherein the first image and the second image include different target objects; Extracting a first object feature from the first image and extracting a second object feature from the second image, wherein the first object feature and the second object feature are object features of the target object in different dimensions; Performing image synthesis based on the first object feature and the second object feature to obtain a third image including a new target object, wherein the new target object has the first object feature and the second object feature; The step of performing image synthesis based on the first object feature and the second object feature to obtain a third image including a new target object comprises: Determine a first object feature reconstruction loss, a second object feature reconstruction loss, and an image reconstruction loss, wherein the first object feature reconstruction loss is a reconstruction loss between a first object feature of the first image and a first object feature of a reconstructed image of the first image, the second object feature reconstruction loss includes a reconstruction loss between a second object feature of the second image and a second object feature of the reconstructed image of the first image, and a reconstruction loss between a second object feature of the second image and a second object feature of the reconstructed image of the second image, and the image reconstruction loss is a loss between a second image and a reconstructed image of the second image; Determining a re-identification loss function based on the first object feature reconstruction loss, the second object feature reconstruction loss, and the image reconstruction loss; Based on a generative adversarial network constrained by the re-identification loss function, the first object feature and the second object feature are synthesized to obtain the third image.

2. The method according to claim 1, wherein: Determine the first object feature reconstruction loss, including: synthesizing the first object feature and the second object feature to obtain a reconstructed image of the first image; extracting a first object feature in a reconstructed image of the first image; A loss between a first object feature extracted in the first image and a first object feature extracted in a reconstructed image of the first image is determined.

3. The method according to claim 1, wherein: The second image includes a first sub-image and a second sub-image, wherein the first sub-image and the second sub-image include the same target object; Determine the second object feature reconstruction loss, including: extracting a second object feature in the first sub-image, and extracting a second object feature in the second sub-image; synthesizing a first object feature extracted from the first image and a second object feature extracted from the first sub-image to obtain a reconstructed image of the first image; extracting a second object feature in a reconstructed image of the first image; A loss between a second object feature extracted in the first sub-image and a second object feature extracted in the second sub-image is determined, and a loss between a second object feature extracted in the first sub-image and a second object feature extracted in a reconstructed image of the first image is determined.

4. The method according to claim 1, wherein: The second image includes a first sub-image and a second sub-image, wherein the first sub-image and the second sub-image include the same target object; Determine the image reconstruction loss, including: extracting a first object feature and a second object feature in the first sub-image, and extracting a second object feature in the second sub-image; synthesizing a first object feature of the first sub-image and a second object feature of the first sub-image to obtain a reconstructed image of the first sub-image; synthesizing the first object feature of the first sub-image and the second object feature of the second sub-image to obtain a reconstructed image of the second sub-image; An image reconstruction loss between the first sub-image and a reconstructed image of the first sub-image is determined, and a reconstruction loss between the second sub-image and a reconstructed image of the second sub-image is determined.

5. An image processing device, comprising: An acquisition module, used for acquiring a first image and a second image, wherein the first image and the second image include different target objects; a determination module, configured to extract a first object feature from the first image and a second object feature from the second image, wherein the first object feature and the second object feature are object features of the target object in different dimensions; a synthesis module, configured to perform image synthesis based on the first object feature and the second object feature to obtain a third image including a new target object, wherein the new target object has the first object feature and the second object feature; Wherein, the synthesis module is also used for: Determine a first object feature reconstruction loss, a second object feature reconstruction loss, and an image reconstruction loss, wherein the first object feature reconstruction loss is a reconstruction loss between a first object feature of the first image and a first object feature of a reconstructed image of the first image; the second object feature reconstruction loss includes a reconstruction loss between a second object feature of the second image and a second object feature of the reconstructed image of the first image, and a reconstruction loss between a second object feature of the second image and a second object feature of the reconstructed image of the second image, and the image reconstruction loss is a loss between a second image and a reconstructed image of the second image; Determining a re-identification loss function based on the first object feature reconstruction loss, the second object feature reconstruction loss, and the image reconstruction loss; Based on a generative adversarial network constrained by the re-identification loss function, the first object feature and the second object feature are synthesized to obtain the third image.

6. The device according to claim 5, wherein: The synthesis module is also used for: synthesizing the first object feature and the second object feature to obtain a reconstructed image of the first image; extracting a first object feature in a reconstructed image of the first image; A loss between a first object feature extracted in the first image and a first object feature extracted in a reconstructed image of the first image is determined.

7. The device according to claim 5, wherein: The second image includes a first sub-image and a second sub-image, wherein the first sub-image and the second sub-image include the same target object; The synthesis module is also used for: extracting a second object feature in the first sub-image, and extracting a second object feature in the second sub-image; synthesizing a first object feature extracted from the first image and a second object feature extracted from the first sub-image to obtain a reconstructed image of the first image; extracting a second object feature in a reconstructed image of the first image; A loss between a second object feature extracted in the first sub-image and a second object feature extracted in the second sub-image is determined, and a loss between a second object feature extracted in the first sub-image and a second object feature extracted in a reconstructed image of the first image is determined.

8. The device according to claim 5, wherein: The second image includes a first sub-image and a second sub-image, wherein the first sub-image and the second sub-image include the same target object; The synthesis module is also used for: extracting a first object feature and a second object feature in the first sub-image, and extracting a second object feature in the second sub-image; synthesizing a first object feature of the first sub-image and a second object feature of the first sub-image to obtain a reconstructed image of the first sub-image; synthesizing the first object feature of the first sub-image and the second object feature of the second sub-image to obtain a reconstructed image of the second sub-image; An image reconstruction loss between the first sub-image and a reconstructed image of the first sub-image is determined, and a reconstruction loss between the second sub-image and a reconstructed image of the second sub-image is determined.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the image processing method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the image processing method according to any one of claims 1 to 4.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the image processing method according to any one of claims 1 to 4.

Images