Image processing method and device

Abstract

The invention relates to an image processing method and device. The method comprises the steps that a to-be-processed picture shot by a camera in virtual shooting is acquired, the to-be-processed picture is obtained by the camera through content shooting based on a virtual scene displayed in a screen and a real scene arranged outside the screen, and the to-be-processed picture comprises a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene; and performing style migration processing on the virtual scene area or the real scene area in the to-be-processed picture by using the style migration model to obtain a processed picture, wherein the real scene area and the virtual scene area in the processed picture have the same image style. According to the invention, for the actual shooting scene in which any virtual scene and real scene are combined to carry out virtual shooting, the picture shot by the camera can be processed, so that the virtual scene and the real scene in the processed picture can be fused together, and the virtual scene and the real scene in the picture are consistent.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0002] Virtual shooting is a shooting method that uses computer-generated images and real-time rendering technology. In virtual shooting, in addition to the virtual scene rendered in real time on the screen, a real scene needs to be arranged in front of the screen to achieve the coordination of virtual and real scenes. Since virtual and real scenes exist at the same time, how to ensure that the virtual scene and the real scene in the picture shot by the camera can be well integrated so that the audience can not easily notice the difference between the two is a crucial issue in virtual shooting. Summary of the invention

[0003] In view of this, the present disclosure proposes an image processing method and device.

[0004] According to one aspect of the present disclosure, there is provided an image processing method for virtual photography, the method comprising:

[0005] Acquire a picture to be processed captured by a camera in virtual shooting, wherein the picture to be processed is obtained by capturing content of a virtual scene displayed on a screen and a real scene arranged outside the screen by the camera, and the picture to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene;

[0006] A style transfer model is used to perform style transfer processing on the virtual scene area or the real scene area in the to-be-processed picture to obtain a processed picture, in which the real scene area and the virtual scene area in the processed picture have the same image style.

[0007] In a possible implementation, the method further includes:

[0008] The style of the picture to be processed is identified by using a style transfer model, a dividing line indicating a change in the image style in the picture to be processed is determined, and a virtual scene area and a real scene area in the picture to be processed are determined based on the dividing line.

[0009] In a possible implementation, using a style transfer model to perform style transfer processing on a virtual scene area or a real scene area in the to-be-processed picture to obtain a processed picture includes:

[0010] The image style of the real scene area in the to-be-processed picture is transferred to the virtual scene area in the to-be-processed picture by using a style transfer model to obtain a processed picture.

[0011] In a possible implementation, the method further includes:

[0012] Acquire multiple sample images;

[0013] Randomly dividing each of the sample images into a first part and a second part;

[0014] The styles of the first parts are changed and then spliced ​​with the second parts of the corresponding sample images to form spliced ​​images corresponding to the sample images;

[0015] Using the dividing lines between different parts in each of the sample images and each of the stitched images as labels of the corresponding stitched images to form a training data set;

[0016] Model training is performed based on the training data set to obtain the style transfer model.

[0017] In a possible implementation manner, randomly dividing each of the sample images into a first part and a second part includes:

[0018] Determining a segmentation range for randomly segmenting the sample image according to a relative position relationship between the virtual scene and the real scene in the image;

[0019] Based on the segmentation range, each of the sample images is randomly segmented into a first part and a second part.

[0020] In a possible implementation manner, the relative positional relationship between the first part and the second part in each of the sample images corresponds to the relative positional relationship between the virtual scene and the real scene in the image.

[0021] In a possible implementation, the method further includes:

[0022] For the first part of each of the sample images, a filter template randomly selected from a plurality of filter templates is used to perform style change.

[0023] According to another aspect of the present disclosure, there is provided an image processing device for virtual photography, the device comprising:

[0024] A picture acquisition module is used to acquire a picture to be processed captured by a camera in virtual shooting, wherein the picture to be processed is obtained by capturing content of a virtual scene displayed on a screen and a real scene arranged outside the screen by the camera, and the picture to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene;

[0025] The picture processing module is used to perform style migration processing on the virtual scene area or the real scene area in the picture to be processed using a style migration model to obtain a processed picture, in which the real scene area and the virtual scene area in the processed picture have the same image style.

[0026] In a possible implementation manner, the device further includes:

[0027] The boundary determination module is used to identify the style of the image to be processed by using a style transfer model, determine a boundary line indicating a change in the image style in the image to be processed, and determine a virtual scene area and a real scene area in the image to be processed based on the boundary line.

[0028] In a possible implementation, using a style transfer model to perform style transfer processing on a virtual scene area or a real scene area in the to-be-processed picture to obtain a processed picture includes:

[0029] The image style of the real scene area in the to-be-processed picture is transferred to the virtual scene area in the to-be-processed picture by using a style transfer model to obtain a processed picture.

[0030] In a possible implementation manner, the device further includes:

[0031] A sample acquisition module, used for acquiring multiple sample images;

[0032] A sample segmentation module, used for randomly segmenting each of the sample images into a first part and a second part;

[0033] An image stitching module, used for stitching each of the first parts with the second part of the corresponding sample image after changing the style of each of the first parts, to form a stitched image corresponding to each of the sample images;

[0034] A data set generation module, used to use the boundary lines between different parts in each of the sample images and each of the stitched images as labels of the corresponding stitched images to form a training data set;

[0035] The model training module is used to perform model training based on the training data set to obtain the style transfer model.

[0036] In a possible implementation manner, randomly dividing each of the sample images into a first part and a second part includes:

[0037] Determining a segmentation range for randomly segmenting the sample image according to a relative position relationship between the virtual scene and the real scene in the image;

[0038] Based on the segmentation range, each of the sample images is randomly segmented into a first part and a second part.

[0039] In a possible implementation manner, the relative positional relationship between the first part and the second part in each of the sample images corresponds to the relative positional relationship between the virtual scene and the real scene in the image.

[0040] In a possible implementation manner, the device further includes:

[0041] The style changing module is used to change the style of the first part of each of the sample images by using a filter template randomly selected from a plurality of filter templates.

[0042] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0043] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0044] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0045] The image processing method and device provided by the embodiments of the present disclosure. For any actual shooting scene in which a virtual scene and a real scene are combined for virtual shooting, the picture shot by the camera can be processed so that the virtual scene and the real scene in the processed picture can be integrated together, and the virtual and real scenes in the picture present a consistent effect. Moreover, the method can be widely used in various application scenarios of virtual shooting, has a wide range of applications, is simple to operate, and does not require the shooting personnel to adjust the lighting and other aspects of the layout of the virtual scene and the real scene, saving the time and labor costs of scene layout.

[0046] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0048] Figure 1 A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0049] Figure 2 A schematic diagram showing an application of an image processing method according to an embodiment of the present disclosure is shown.

[0050] Figure 3 A schematic diagram showing a processing process of a sample image in an image processing method according to an embodiment of the present disclosure is shown.

[0051] Figure 4 A schematic diagram showing model training in an image processing method according to an embodiment of the present disclosure.

[0052] Figure 5 is a block diagram of a device 1900 for image processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0053] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0054] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0055] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0056] In order to make the virtual scene and the real scene in the picture taken by the camera blend well together, so that the audience will not easily notice the difference between the two. In the related art, the virtual scene and / or the real scene are adjusted by lighting, etc., so that the virtual scene and the real scene in the captured picture can be blended together. However, such an adjustment method is complicated to operate. Moreover, targeted adjustment operations are required for different combinations of virtual and real scenes, which has limitations and increases the cost of adjustment.

[0057] In order to solve the above technical problems, the embodiments of the present disclosure provide an image processing method and device. For an actual shooting scene in which any virtual scene and real scene are combined for virtual shooting, the picture shot by the camera can be processed so that the virtual scene and the real scene in the processed picture can be integrated together, and the virtual and real scenes in the picture present a consistent effect. Moreover, the method can be widely used in various application scenarios of virtual shooting, with a wide range of applications and simple operation, and there is no need for the shooting personnel to adjust the lighting and other aspects of the layout of the virtual scene and the real scene, saving the time and labor costs of scene layout.

[0058] like Figure 1 , Figure 2As shown, the image processing method provided by the embodiment of the present disclosure includes step S101-step S102. The method can be applied to virtual shooting, and the method can be executed by an electronic device or a server.

[0059] In step S101, a picture to be processed captured by a camera in virtual shooting is obtained, wherein the picture to be processed is obtained by capturing content of a virtual scene displayed on a screen and a real scene arranged outside the screen by the camera, and the picture to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene.

[0060] In this embodiment, during the virtual shooting process, the virtual scene may refer to a scene rendered based on shooting requirements and displayed on the screen. The real scene may be a scene arranged by placing corresponding objects around the screen based on shooting requirements. The real scene and the virtual scene displayed on the screen together constitute the actual shooting scene of the virtual shooting. In the process of virtual shooting using a camera, the picture to be processed may be an image captured by the camera, or each video frame of the video captured by the camera. The picture to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene, wherein since the active objects such as actors that can move are in the real scene, the active objects such as actors in the picture to be processed are naturally also in the real scene area. Therefore, in the embodiment of the present disclosure, the real scene area in the picture to be processed may be formed by shooting a real scene including fixed objects and active objects that are moving. The part of the picture to be processed except the real scene area is the virtual scene area, that is, the virtual scene area is formed by shooting the part of the virtual scene displayed on the screen that is not blocked by real objects such as actors. Due to the differences between virtual scenes and real scenes, there are differences in image style between the real scene area and the virtual scene area in the captured image to be processed. The image style can be represented by parameters such as brightness, hue, contrast, color saturation, and gamma transform.

[0061] In step S102, a style transfer model is used to perform style transfer processing on the virtual scene area or the real scene area in the to-be-processed picture to obtain a processed picture, in which the real scene area and the virtual scene area in the processed picture have the same image style.

[0062] In this embodiment, the style transfer model is pre-trained. After the image to be processed is input into the style transfer model, the style transfer model will first identify the image to be processed and determine the real scene area and the virtual scene area in the model. Then the style transfer model will perform style transfer processing on the virtual scene area or the real scene area, so that the real scene area and the virtual scene area have the same image style to obtain a processed image.

[0063] In a possible implementation, the method may further include: using a style transfer model to identify the style of the picture to be processed, determining a dividing line indicating a change in image style in the picture to be processed, and determining a virtual scene area and a real scene area in the picture to be processed based on the dividing line.

[0064] In this implementation, since the virtual scene area and the real scene area in the to-be-processed picture obtained after shooting the virtual scene and the real scene have obvious differences in image style such as brightness, hue, contrast, color saturation, and gamma transformation, the style of the to-be-processed picture can be identified and analyzed using a style transfer model, and the dividing line caused by the change in image style caused by the difference between the virtual and real scenes in the to-be-processed picture can be determined, and then the virtual scene area and the real scene area in the to-be-processed picture can be determined in combination with the dividing line. The dividing line can be a straight line or a curve, and can be one or more. The shape and number of the dividing lines are related to the relative positional relationship between the virtual scene area and the real scene area in the to-be-processed picture, and the present disclosure does not limit the dividing line.

[0065] In some embodiments, using a style transfer model to perform style transfer processing on the virtual scene area or the real scene area in the picture to be processed to obtain a processed picture may include: using a style transfer model to transfer the image style of the real scene area in the picture to be processed to the virtual scene area in the picture to be processed to obtain a processed picture. That is, the style transfer model can be used to adjust the image style of the virtual scene area in the picture to be processed to be consistent with the image style of the real scene area in the picture to be processed to obtain a processed picture. Since the virtual area corresponds to a virtual scene, and the virtual scene itself is generated by rendering, it is more convenient and easy to adjust the image style of the virtual scene area compared to the real scene area corresponding to the real scene, so that the image style of the real scene area is transferred to the virtual scene area more efficiently and faster.

[0066] In a possible implementation, in order to make the style transfer model more effective, the method may further include: obtaining a plurality of sample images; randomly dividing each of the sample images into a first part and a second part; performing style changes on each of the first parts and splicing them with the second part of the corresponding sample image to form a spliced ​​image corresponding to each of the sample images (e.g. Figure 3 As shown in the figure); taking the dividing lines between different parts in each of the sample images and each of the stitched images (that is, the dividing lines between the first part and the second part after the style change in each of the stitched images) as labels of the corresponding stitched images to form a training data set; performing model training based on the training data set to obtain the style transfer model.

[0067] In this implementation, the sample image may be a landscape image or other image related to virtual photography, which is not limited in the present disclosure. In order to further improve the accuracy of the style transfer model, the sample image may be as similar or related as possible to the actual shooting scene of the virtual photography to be performed.

[0068] In a possible implementation, randomly dividing each of the sample images into a first part and a second part may include: determining a segmentation range for randomly dividing the sample image according to the relative positional relationship between the virtual scene and the real scene in the image; and randomly dividing each of the sample images into a first part and a second part based on the segmentation range. The segmentation range may refer to the area range where the segmentation line for random segmentation in the sample image is located, and the segmentation range may be one or more. The segmentation line may be a horizontal line, a curve, etc. within the segmentation range. The "random" here may refer to randomly determining the specific position and shape of the segmentation line within the segmentation range. Since the relative positional relationship between the virtual scene and the real scene is different in different virtual shooting tasks, the segmentation range of the sample image may be set based on the relative positional relationship between the virtual scene and the real scene in different virtual shooting tasks. In this way, corresponding style transfer models may be trained for different virtual shooting tasks to meet the shooting requirements of different virtual shooting tasks. For example, generally speaking, the virtual scene displayed on the screen is located in the upper part of the image, and the real scene is located in the lower middle part of the image. For a certain shooting scene, the ratio between the size of the screen and the size of the range occupied by the real scene built in front of the screen can determine the basic ratio between the first part and the second part. Since the camera shooting angle is different, this basic ratio will change in the image. Therefore, a ratio range can be determined around the basic ratio based on the camera shooting angle range. In addition to fixed objects, there are also active objects such as actors who can move in the real scene. These objects will cause the boundaries of the real scene to change, so the object range can also be set for these active objects. For example, considering the occlusion of the screen by the actors, the segmentation range can be further expanded upwards, and finally the possible segmentation range can be determined by combining the ratio range and the object range. For example, Figure 3 As shown, the first part I of the sample image I up and Part II I down can be the upper and lower halves of the sample image (for simplicity) Figure 3 , Figure 4 In the figure, only an example of a simple division of the sample image into two parts, the upper and lower parts, is given schematically) to correspond to the virtual scene at the top and the virtual scene at the bottom in the actual shooting scene of the virtual shooting, that is, the relative position relationship between the first part and the second part in each of the sample images corresponds to the relative position relationship between the virtual scene and the real scene in the image, thereby improving the accuracy of the model.

[0069] In some embodiments, the first part I of each sample image may be up , using a randomly selected filter template from multiple filter templates to change the style, and obtain the changed first part I u ′ p , so that the first part I of each sample image after the change u ′ p Part II down The style is inconsistent, and finally the first part I u ′ p Corresponding to Part II I down The stitching image I' is obtained after stitching. Among them, multiple filter templates can be generated in advance according to the needs, and then in the process of processing the first part of each sample image, a filter template is randomly selected to change the style of the first part of the sample image. u ′ p , corresponding to the virtual scene, multiple filter templates can enable the image to present the color, brightness and other image features of the image part where the virtual scene is located in the picture captured by the camera after the filter template is applied.

[0070] In this embodiment, in the training data set formed, each stitched image I' is used as the input of the model, the sample image I corresponding to the stitched image I', and the boundary between different parts of the stitched image I' are used as the label of the stitched image. Among them, a deep learning model can be constructed based on the Encoder-Decoder architecture as the initial model for training to obtain a style transfer model after the training is completed. The initial model can also be constructed using other models, and the present disclosure does not limit this. In some embodiments, such as Figure 4 As shown, the initial model may include a feature extraction module and an image reconstruction module. The feature extraction module may extract features from the input image and input the extracted image features into the image reconstruction module; the image reconstruction module may reconstruct the input image based on the image features, transfer the style of one of two parts with different styles in the image to the other part, and finally output an image with consistent style. Image features may include context features (used to describe the relationship and context information between different regions in the image), color features, texture features, etc., which are not limited in the present disclosure.

[0071] like Figure 4As shown, during the model training process, the spliced ​​image and the mask (mark) of the spliced ​​image are used as input, and then the feature extraction module can extract image features from the spliced ​​image, and the image reconstruction module can combine the image features and the mask to reconstruct the image, generate a predicted image with consistent style, and output an image segmentation result for style segmentation of the spliced ​​image. The image segmentation result can be the first part I of the spliced ​​image I' after the change u ′ p Part II down The boundary between the regions. Then, based on a predetermined model training strategy, the predicted image, the image segmentation result, the corresponding sample image, and the boundary between different parts in each of the stitched images are combined to update and adjust the parameters of the model until the model converges to obtain a style transfer model. Among them, the model training strategy includes the loss function used for model training and the optimization algorithm for parameter update (such as stochastic gradient descent (SGD)). The optimization goal may include making the style of the changed first part and the second part as close as possible, the content of the changed first part as close as possible to the original first part, and the boundary between the regions as close as possible to the boundary in the label. Among them, the mask of the spliced ​​image can be an image formed by occluding the spliced ​​image. The mask is used to cover the second part of the spliced ​​image in the model training to ensure that the second part will not be affected by the style conversion in the predicted image output in the end, and the original style remains unchanged.

[0072] In some embodiments, in order to improve the performance of the model, data enhancement techniques such as rotation, translation, and scaling may be used to process the stitched image before inputting it into the model to enhance the robustness of the model for detecting images of different perspectives and scales.

[0073] In a possible implementation, the method may further include: displaying the processed picture. After obtaining the processed picture, the picture to be processed may be displayed locally or using other display devices. Wherein, in the case where the picture to be processed is a video obtained by virtual shooting, what is displayed to the user may be a processed video formed based on the processed picture.

[0074] In this embodiment, the monitoring equipment of the virtual shooting scene can be used to display the processed pictures to the user, so that the staff on the scene can understand the shooting situation based on the processed pictures displayed in real time in the monitoring equipment. In some embodiments, the terminal such as a mobile phone can also be controlled to display the processed pictures to the user.

[0075] The present disclosure also provides an image processing device for virtual photography, the device comprising:

[0076] A picture acquisition module is used to acquire a picture to be processed captured by a camera in virtual shooting, wherein the picture to be processed is obtained by capturing content of a virtual scene displayed on a screen and a real scene arranged outside the screen by the camera, and the picture to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene;

[0077] The picture processing module is used to perform style migration processing on the virtual scene area or the real scene area in the picture to be processed using a style migration model to obtain a processed picture, in which the real scene area and the virtual scene area in the processed picture have the same image style.

[0078] In a possible implementation manner, the device further includes:

[0079] The boundary determination module is used to identify the style of the image to be processed by using a style transfer model, determine a boundary line indicating a change in the image style in the image to be processed, and determine a virtual scene area and a real scene area in the image to be processed based on the boundary line.

[0080] In a possible implementation, using a style transfer model to perform style transfer processing on a virtual scene area or a real scene area in the to-be-processed picture to obtain a processed picture includes:

[0081] The image style of the real scene area in the to-be-processed picture is transferred to the virtual scene area in the to-be-processed picture by using a style transfer model to obtain a processed picture.

[0082] In a possible implementation manner, the device further includes:

[0083] A sample acquisition module, used for acquiring multiple sample images;

[0084] A sample segmentation module, used for randomly segmenting each of the sample images into a first part and a second part;

[0085] An image stitching module, used for stitching each of the first parts with the second part of the corresponding sample image after changing the style of each of the first parts, to form a stitched image corresponding to each of the sample images;

[0086] A data set generation module, used to use the boundary lines between different parts in each of the sample images and each of the stitched images as labels of the corresponding stitched images to form a training data set;

[0087] The model training module is used to perform model training based on the training data set to obtain the style transfer model.

[0088] In a possible implementation manner, randomly dividing each of the sample images into a first part and a second part includes:

[0089] Determining a segmentation range for randomly segmenting the sample image according to a relative position relationship between the virtual scene and the real scene in the image;

[0090] Based on the segmentation range, each of the sample images is randomly segmented into a first part and a second part.

[0091] In a possible implementation manner, the relative positional relationship between the first part and the second part in each of the sample images corresponds to the relative positional relationship between the virtual scene and the real scene in the image.

[0092] In a possible implementation manner, the device further includes:

[0093] The style changing module is used to change the style of the first part of each of the sample images by using a filter template randomly selected from a plurality of filter templates.

[0094] It should be noted that although the above embodiments are used as examples to introduce the image processing method and device, those skilled in the art will appreciate that the present disclosure should not be limited thereto. In fact, users can flexibly set each step and module according to personal preferences and / or actual application scenarios, as long as they comply with the technical solution of the present disclosure.

[0095] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0096] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0097] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0098] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0099] Figure 51 is a block diagram of a device 1900 for image processing according to an exemplary embodiment. For example, the device 1900 may be provided as a server or an electronic device. Figure 5 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0100] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0101] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0102] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0103] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0104] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0105] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0106] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0109] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0110] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that: Applied to virtual shooting, the method includes: Acquire a picture to be processed captured by a camera in virtual shooting, wherein the picture to be processed is obtained by capturing content of a virtual scene displayed on a screen and a real scene arranged outside the screen by the camera, and the picture to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene; A style transfer model is used to perform style transfer processing on the virtual scene area or the real scene area in the to-be-processed picture to obtain a processed picture, in which the real scene area and the virtual scene area in the processed picture have the same image style.

2. The method according to claim 1, characterized in that: The method further comprises: The style of the picture to be processed is identified by using a style transfer model, a dividing line indicating a change in the image style in the picture to be processed is determined, and a virtual scene area and a real scene area in the picture to be processed are determined based on the dividing line.

3. The method according to claim 1, characterized in that Using a style transfer model to perform style transfer processing on a virtual scene area or a real scene area in the to-be-processed picture to obtain a processed picture, including: The image style of the real scene area in the to-be-processed picture is transferred to the virtual scene area in the to-be-processed picture by using a style transfer model to obtain a processed picture.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire multiple sample images; Randomly dividing each of the sample images into a first part and a second part; The styles of the first parts are changed and then spliced ​​with the second parts of the corresponding sample images to form spliced ​​images corresponding to the sample images; Using the dividing lines between different parts in each of the sample images and each of the stitched images as labels of the corresponding stitched images to form a training data set; Model training is performed based on the training data set to obtain the style transfer model.

5. The method according to claim 4, characterized in that Randomly dividing each of the sample images into a first part and a second part, comprising: Determining a segmentation range for randomly segmenting the sample image according to a relative position relationship between the virtual scene and the real scene in the image; Based on the segmentation range, each of the sample images is randomly segmented into a first part and a second part.

6. The method according to claim 5, characterized in that The relative positional relationship between the first part and the second part in each of the sample images corresponds to the relative positional relationship between the virtual scene and the real scene in the image.

7. The method according to claim 5, characterized in that The method further comprises: For the first part of each of the sample images, a filter template randomly selected from a plurality of filter templates is used to perform style change.

8. An image processing device, characterized in that: Applied to virtual photography, the device comprises: A picture acquisition module is used to acquire a picture to be processed captured by a camera in virtual shooting, wherein the picture to be processed is obtained by capturing content of a virtual scene displayed on a screen and a real scene arranged outside the screen by the camera, and the picture to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene; The picture processing module is used to perform style migration processing on the virtual scene area or the real scene area in the picture to be processed using a style migration model to obtain a processed picture, in which the real scene area and the virtual scene area in the processed picture have the same image style.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, characterized in that: When the computer readable code runs in a processor of an electronic device, the processor in the electronic device executes the method according to any one of claims 1 to 7.

Images