Image processing method and apparatus

Abstract

This disclosure relates to an image processing method and apparatus. The method includes: acquiring a scene to be processed captured by a camera during virtual shooting, the scene to be processed being captured by the camera based on a virtual scene displayed on a screen and a real scene set up outside the screen, the scene to be processed including a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene; using a style transfer model to perform style transfer processing on the virtual scene area or the real scene area in the scene to be processed to obtain a processed image, in which the real scene area and the virtual scene area have the same image style. For any actual shooting scenario combining virtual and real scenes for virtual shooting, it can realize the processing of the scene captured by the camera, so that the virtual scene and the real scene in the processed image can be merged together, and the virtual and real scenes in the image present a consistent effect.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method and apparatus. Background Technology

[0002] Virtual filming is a shooting method that utilizes computer-generated images and real-time rendering technology. In virtual filming, in addition to the virtual scene rendered in real-time on the screen, a real-world scene needs to be set up in front of the screen to achieve a seamless integration of the virtual and real scenes. Because the virtual and real scenes coexist, ensuring that the virtual and real scenes blend seamlessly in the footage captured by the camera, making it difficult for viewers to perceive any difference, is a crucial issue in virtual filming. Summary of the Invention

[0003] In view of this, the present disclosure proposes an image processing method and apparatus.

[0004] According to one aspect of this disclosure, an image processing method is provided for virtual photography, the method comprising:

[0005] The image to be processed is captured by the camera in virtual shooting. The image to be processed is obtained by the camera capturing content based on the virtual scene displayed on the screen and the real scene set up outside the screen. The image to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene.

[0006] A style transfer model is used to perform style transfer processing on the virtual or real scene areas in the image to be processed, resulting in a processed image in which the real scene areas and virtual scene areas have the same image style.

[0007] In one possible implementation, the method further includes:

[0008] The style of the image to be processed is identified using a style transfer model, the boundary line indicating the style change in the image to be processed is determined, and the virtual scene area and the real scene area in the image to be processed are determined based on the boundary line.

[0009] In one possible implementation, a style transfer model is used to perform style transfer processing on the virtual or real scene regions in the image to be processed, resulting in a processed image, including:

[0010] The style transfer model is used to transfer the image style of the real scene area in the image to the virtual scene area in the image to be processed, resulting in the processed image.

[0011] In one possible implementation, the method further includes:

[0012] Acquire multiple sample images;

[0013] Each of the sample images is randomly divided into a first part and a second part;

[0014] After the style of each first part is changed, it is stitched together with the second part of the corresponding sample image to form a stitched image corresponding to each sample image;

[0015] The boundary lines between different parts of each sample image and each stitched image are used as the labels of the corresponding stitched images to form a training dataset.

[0016] The style transfer model is obtained by training the model based on the training dataset.

[0017] In one possible implementation, each of the sample images is randomly divided into a first part and a second part, including:

[0018] Based on the relative positional relationship between the virtual scene and the real scene in the image, the segmentation range for randomly segmenting the sample image is determined;

[0019] Based on the segmentation range, each of the sample images is randomly segmented into a first part and a second part.

[0020] In one possible implementation, the relative positional relationship between the first part and the second part in each of the sample images corresponds to the relative positional relationship between the virtual scene and the real scene in the image.

[0021] In one possible implementation, the method further includes:

[0022] For the first part of each of the sample images, a style change is performed using a filter template randomly selected from multiple filter templates.

[0023] According to another aspect of this disclosure, an image processing apparatus is provided for virtual shooting, the apparatus comprising:

[0024] The image acquisition module is used to acquire the image to be processed captured by the camera in virtual shooting. The image to be processed is obtained by the camera capturing content based on the virtual scene displayed on the screen and the real scene set up outside the screen. The image to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene.

[0025] The image processing module is used to perform style transfer processing on the virtual or real scene areas in the image to be processed using a style transfer model to obtain a processed image, wherein the real scene areas and virtual scene areas in the processed image have the same image style.

[0026] In one possible implementation, the device further includes:

[0027] The boundary determination module is used to identify the style of the image to be processed using a style transfer model, determine the boundary line indicating the style change of the image in the image to be processed, and determine the virtual scene area and the real scene area in the image to be processed based on the boundary line.

[0028] In one possible implementation, a style transfer model is used to perform style transfer processing on the virtual or real scene regions in the image to be processed, resulting in a processed image, including:

[0029] The style transfer model is used to transfer the image style of the real scene area in the image to the virtual scene area in the image to be processed, resulting in the processed image.

[0030] In one possible implementation, the device further includes:

[0031] The sample acquisition module is used to acquire multiple sample images;

[0032] The sample segmentation module is used to randomly segment each of the sample images into a first part and a second part;

[0033] The image stitching module is used to stitch the first part of each image with the second part of the corresponding sample image after style modification, so as to form a stitched image corresponding to each sample image.

[0034] The dataset generation module is used to take the boundary lines between different parts of each sample image and each stitched image as the labels of the corresponding stitched images to form a training dataset.

[0035] The model training module is used to train the model based on the training dataset to obtain the style transfer model.

[0036] In one possible implementation, each of the sample images is randomly divided into a first part and a second part, including:

[0037] Based on the relative positional relationship between the virtual scene and the real scene in the image, the segmentation range for randomly segmenting the sample image is determined;

[0038] Based on the segmentation range, each of the sample images is randomly segmented into a first part and a second part.

[0039] In one possible implementation, the relative positional relationship between the first part and the second part in each of the sample images corresponds to the relative positional relationship between the virtual scene and the real scene in the image.

[0040] In one possible implementation, the device further includes:

[0041] The style change module is used to change the style of the first part of each of the sample images by using a filter template randomly selected from multiple filter templates.

[0042] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing instructions stored in the memory.

[0043] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0044] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0045] The image processing method and apparatus disclosed herein can process images captured by a camera in any virtual shooting scenario combining virtual and real scenes. This allows the virtual and real scenes to be seamlessly integrated into the processed image, resulting in a consistent visual effect. Furthermore, this method is widely applicable to various virtual shooting scenarios, offering broad applicability and ease of operation. It eliminates the need for personnel to adjust lighting or other aspects of the virtual and real scenes, saving time and manpower costs associated with scene setup.

[0046] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0047] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0048] Figure 1 A flowchart illustrating an image processing method according to an embodiment of the present disclosure is shown.

[0049] Figure 2 A schematic diagram illustrating the application of an image processing method according to an embodiment of the present disclosure is shown.

[0050] Figure 3 This diagram illustrates the processing procedure of a sample image in an image processing method according to an embodiment of the present disclosure.

[0051] Figure 4 A schematic diagram of model training in an image processing method according to an embodiment of the present disclosure is shown.

[0052] Figure 5 This is a block diagram illustrating an apparatus 1900 for image processing according to an exemplary embodiment. Detailed Implementation

[0053] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0054] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0055] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0056] To seamlessly blend virtual and real-world scenes captured by a camera, making the differences imperceptible to viewers, related technologies involve adjusting lighting and other aspects of the virtual and / or real-world scenes to achieve a harmonious integration. However, this method is complex and requires tailored adjustments for different scenarios, limiting its effectiveness and increasing costs.

[0057] To address the aforementioned technical problems, this disclosure provides an image processing method and apparatus. For any actual shooting scenario combining virtual and real scenes for virtual filming, this method can process the footage captured by the camera, enabling the virtual and real scenes to be seamlessly integrated in the processed image, resulting in a consistent visual effect. Furthermore, this method can be widely applied to various virtual filming scenarios, offering broad applicability, simple operation, and eliminating the need for personnel to adjust lighting or other aspects of the virtual and real scene setup, thus saving time and manpower costs associated with scene arrangement.

[0058] like Figure 1 , Figure 2As shown, the image processing method provided in this embodiment includes steps S101-S102. This method can be applied to virtual shooting, and the method can be executed by an electronic device or a server.

[0059] In step S101, the image to be processed captured by the camera during virtual shooting is obtained. The image to be processed is obtained by the camera capturing content based on the virtual scene displayed on the screen and the real scene set up outside the screen. The image to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene.

[0060] In this embodiment, during virtual shooting, the virtual scene can refer to a scene rendered on the screen based on shooting requirements. The real scene can be a scene arranged around the screen by placing corresponding objects based on shooting requirements. The real scene and the virtual scene displayed on the screen together constitute the actual shooting scene for virtual shooting. During virtual shooting using a camera, the image to be processed can be an image captured by the camera or each video frame of a video captured by the camera. The image to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene. Since actors and other movable objects are in the real scene, the actors and other movable objects in the image to be processed are naturally also in the real scene area. Therefore, in this embodiment, the real scene area in the image to be processed can be formed by shooting the real scene including fixed objects and movable objects. The part of the image to be processed other than the real scene area is the virtual scene area, that is, the virtual scene area is formed by shooting the part of the virtual scene displayed on the screen that is not obscured by actors and other real objects. Because of the differences between virtual and real scenes, the real and virtual areas in the captured image to be processed will have different image styles. Image style can be represented by parameters such as brightness, hue, contrast, color saturation, and gamma transformation.

[0061] In step S102, a style transfer model is used to perform style transfer processing on the virtual or real scene areas in the image to be processed to obtain a processed image, in which the real scene areas and virtual scene areas have the same image style.

[0062] In this embodiment, the style transfer model is pre-trained. After the image to be processed is input into the style transfer model, the style transfer model will first identify the image to be processed, determine the real scene area and the virtual scene area in the model, and then the style transfer model will perform style transfer processing on the virtual scene area or the real scene area, so that the real scene area and the virtual scene area have the same image style, and obtain the processed image.

[0063] In one possible implementation, the method may further include: using a style transfer model to identify the style of the image to be processed, determining a boundary line indicating the style change in the image to be processed, and determining the virtual scene area and the real scene area in the image to be processed based on the boundary line.

[0064] In this implementation, since the virtual and real scene regions in the image to be processed differ significantly in image style (brightness, hue, contrast, color saturation, gamma transformation, etc.) after capturing virtual and real scenes, a style transfer model can be used to identify and analyze the style of the image to be processed. This allows for the determination of the boundary line resulting from the difference in image style between virtual and real scenes. Then, by combining this boundary line, the virtual and real scene regions in the image to be processed can be identified. The boundary line can be a straight line, a curve, or one or more lines. The shape and number of boundary lines are related to the relative positional relationship between the virtual and real scene regions in the image to be processed; this disclosure does not impose any restrictions on the boundary line.

[0065] In some embodiments, using a style transfer model to perform style transfer processing on the virtual or real scene regions in the image to be processed to obtain a processed image may include: using a style transfer model to transfer the image style of the real scene region in the image to be processed to the virtual scene region in the image to be processed, thereby obtaining a processed image. That is, a style transfer model can be used to adjust the image style of the virtual scene region in the image to be processed to be consistent with the image style of the real scene region in the image to be processed, thereby obtaining a processed image. Since the virtual region corresponds to a virtual scene, and the virtual scene itself is rendered and generated, adjusting the image style of the virtual scene region is more convenient and easier than adjusting the image style of the real scene region compared to the real scene region corresponding to the real scene, making the transfer of the image style of the real scene region to the virtual scene region more efficient and faster.

[0066] In one possible implementation, to improve the performance of the style transfer model, the method may further include: acquiring multiple sample images; randomly dividing each sample image into a first part and a second part; performing style changes on each first part and then stitching it with the corresponding second part of the sample image to form a stitched image corresponding to each sample image (e.g., ...). Figure 3 (as shown); the dividing lines between different parts of each sample image and each stitched image (that is, the dividing lines between the first part and the second part after style change in each stitched image) are used as the labels of the corresponding stitched images to form a training dataset; the model is trained based on the training dataset to obtain the style transfer model.

[0067] In this implementation, the sample images can be landscape images or other images related to virtual shooting, and this disclosure does not impose any limitations on this. To further improve the accuracy of the style transfer model, the sample images can be as similar as possible to or related to the actual shooting scene of the virtual shooting to be carried out.

[0068] In one possible implementation, randomly segmenting each sample image into a first part and a second part may include: determining a segmentation range for randomly segmenting the sample image based on the relative positional relationship between the virtual scene and the real scene in the image; and randomly segmenting each sample image into a first part and a second part based on the segmentation range. The segmentation range can refer to the area where the segmentation line for random segmentation is located in the sample image, and there can be one or more segmentation ranges. The segmentation line can be a horizontal line, a curve, etc., within the segmentation range. Here, "random" can mean randomly determining the specific position and shape of the segmentation line within the segmentation range. Since the relative positional relationship between the virtual scene and the real scene differs in different virtual shooting tasks, the segmentation range of the sample image can be set based on the relative positional relationship between the virtual scene and the real scene in different virtual shooting tasks. This allows for the training of corresponding style transfer models for different virtual shooting tasks, meeting the shooting requirements of different virtual shooting tasks. For example, generally, the virtual scene displayed on the screen is located at the upper part of the image, and the real scene is located at the lower part. For a given shooting scene, the ratio between the size of the screen and the area occupied by the real scene built in front of the screen can determine the basic ratio between the first and second parts. However, due to different camera shooting angles, this basic ratio will change in the image. Therefore, a ratio range can be determined around the basic ratio based on the camera's shooting angle. Since the real scene contains not only stationary objects but also moving objects such as actors, these objects will change the boundaries of the real scene. Therefore, object ranges can also be set for these moving objects. For example, considering the actors' obstruction of the screen, the segmentation range can be further expanded upwards. Finally, combining the ratio range and the object range, a possible segmentation range can be determined. For example, as... Figure 3 As shown, the first part I of sample image I up Part I down These can be the upper and lower halves of the sample image, respectively (for simplicity) Figure 3 , Figure 4 The example shown is only schematically illustrated by dividing the sample image into two parts (upper and lower) to correspond to the virtual scene being above and below in the actual shooting scene of the virtual shooting. This means that the relative positional relationship between the first part and the second part in each sample image is obtained, which corresponds to the relative positional relationship between the virtual scene and the real scene in the image, thereby improving the accuracy of the model.

[0069] In some embodiments, the first portion I of each of the sample images can be... up The style of the first part I is obtained by randomly selecting a filter template from multiple filter templates and changing its style. u ′ p So that the first part I after the change of each sample image u ′ p With Part I down The styles were inconsistent, ultimately resulting in the revised first part I. u ′ p With the corresponding second part I down The stitched image I' is obtained after stitching. Multiple filter templates can be pre-generated as needed, and then, during the processing of the first part of each sample image, a filter template is randomly selected to style the first part of the sample image. If the first part I after applying the filter template is... u ′ p For virtual scenes, multiple filter templates can enable the image to present the color, brightness, and other image features of the image part of the virtual scene captured by the camera after the filter template is applied.

[0070] In this embodiment, in the formed training dataset, each stitched image I' serves as the input to the model, the corresponding sample image I is used, and the boundary lines between different parts of the stitched image I' serve as the label for that stitched image. A deep learning model can be constructed based on an Encoder-Decoder architecture as the initial model for training, to obtain a style transfer model after training. Alternatively, other models can be used to construct the initial model; this disclosure does not limit this approach. In some embodiments, such as... Figure 4 As shown, the initial model may include a feature extraction module and an image reconstruction module. The feature extraction module can extract features from the input image and input the extracted image features into the image reconstruction module; the image reconstruction module can reconstruct the input image based on the image features, transferring the style of one part of two parts with different styles to the other part, and finally outputting an image with a consistent style. Image features may include contextual features (used to describe the relationship and contextual information between different regions in the image), color features, texture features, etc., and this disclosure does not limit them.

[0071] like Figure 4As shown, during model training, the stitched image and its mask are taken as input. The feature extraction module then extracts image features from the stitched image, and the image reconstruction module combines these features with the mask to reconstruct the image, generating a style-consistent predicted image and outputting the style-classified image segmentation result for the stitched image. The image segmentation result can be the modified first part I of the stitched image I'. u ′ p With Part I down The boundary lines between the regions are defined. Then, based on a predetermined model training strategy, the model parameters are updated and adjusted using the predicted image, image segmentation results, corresponding sample images, and the boundary lines between different parts of each stitched image, until the model converges to obtain the style transfer model. The model training strategy includes the loss function used for model training and the optimization algorithm for parameter updates (such as Stochastic Gradient Descent (SGD)). Optimization objectives may include making the styles of the changed first and second parts as close as possible, the content of the changed first part as close as possible to the original first part, and the boundary lines between regions as close as possible to the boundary lines in the labels. The mask for the stitched image can be an image formed by occluding the stitched image. During model training, a mask is used to occlude the second part of the stitched image, ensuring that the second part is not affected by style transfer in the final output predicted image, maintaining its original style.

[0072] In some embodiments, to improve the performance of the model, data augmentation techniques such as rotation, translation, and scaling can be used to process the stitched image before inputting it into the model, thereby enhancing the model's robustness to image detection at different viewpoints and scales.

[0073] In one possible implementation, the method may further include: displaying the processed image. After obtaining the processed image, the image to be processed can be displayed locally or using other display devices. Wherein, if the image to be processed is a virtually captured video, the video displayed to the user can be a processed video formed based on the processed image.

[0074] In this embodiment, the processed footage can be displayed to the user using virtual shooting location monitoring equipment, allowing on-site staff to understand the shooting situation based on the real-time display of the processed footage on the monitoring equipment. In some embodiments, a mobile phone or other terminal can also be controlled to display the processed footage to the user.

[0075] This disclosure also provides an image processing apparatus for virtual shooting, the apparatus comprising:

[0076] The image acquisition module is used to acquire the image to be processed captured by the camera in virtual shooting. The image to be processed is obtained by the camera capturing content based on the virtual scene displayed on the screen and the real scene set up outside the screen. The image to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene.

[0077] The image processing module is used to perform style transfer processing on the virtual or real scene areas in the image to be processed using a style transfer model to obtain a processed image, wherein the real scene areas and virtual scene areas in the processed image have the same image style.

[0078] In one possible implementation, the device further includes:

[0079] The boundary determination module is used to identify the style of the image to be processed using a style transfer model, determine the boundary line indicating the style change of the image in the image to be processed, and determine the virtual scene area and the real scene area in the image to be processed based on the boundary line.

[0080] In one possible implementation, a style transfer model is used to perform style transfer processing on the virtual or real scene regions in the image to be processed, resulting in a processed image, including:

[0081] The style transfer model is used to transfer the image style of the real scene area in the image to the virtual scene area in the image to be processed, resulting in the processed image.

[0082] In one possible implementation, the device further includes:

[0083] The sample acquisition module is used to acquire multiple sample images;

[0084] The sample segmentation module is used to randomly segment each of the sample images into a first part and a second part;

[0085] The image stitching module is used to stitch the first part of each image with the second part of the corresponding sample image after style modification, so as to form a stitched image corresponding to each sample image.

[0086] The dataset generation module is used to take the boundary lines between different parts of each sample image and each stitched image as the labels of the corresponding stitched images to form a training dataset.

[0087] The model training module is used to train the model based on the training dataset to obtain the style transfer model.

[0088] In one possible implementation, each of the sample images is randomly divided into a first part and a second part, including:

[0089] Based on the relative positional relationship between the virtual scene and the real scene in the image, the segmentation range for randomly segmenting the sample image is determined;

[0090] Based on the segmentation range, each of the sample images is randomly segmented into a first part and a second part.

[0091] In one possible implementation, the relative positional relationship between the first part and the second part in each of the sample images corresponds to the relative positional relationship between the virtual scene and the real scene in the image.

[0092] In one possible implementation, the device further includes:

[0093] The style change module is used to change the style of the first part of each of the sample images by using a filter template randomly selected from multiple filter templates.

[0094] It should be noted that although the image processing method and apparatus have been described above as examples, those skilled in the art will understand that this disclosure is not limited thereto. In fact, users can flexibly set each step and module according to their personal preferences and / or actual application scenarios, as long as it conforms to the technical solution of this disclosure.

[0095] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0096] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0097] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0098] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0099] Figure 5This is a block diagram illustrating an apparatus 1900 for image processing according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or electronic device. (Refer to...) Figure 5 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0100] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0101] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0102] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0103] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0104] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0105] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0106] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0108] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0110] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that, The method, applied to virtual shooting, includes: The image to be processed is captured by the camera in virtual shooting. The image to be processed is obtained by the camera capturing content based on the virtual scene displayed on the screen and the real scene set up outside the screen. The real scene refers to the scene set up around the screen by placing objects based on shooting needs. The image to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene. A style transfer model is used to perform style transfer processing on the virtual or real scene areas in the image to be processed to obtain a processed image, in which the real scene areas and the virtual scene areas have the same image style. The method further includes: using a style transfer model to identify the style of the image to be processed, determining the boundary line indicating the change in image style in the image to be processed, and determining the virtual scene area and the real scene area in the image to be processed based on the boundary line; The method further includes: acquiring multiple sample images; determining a segmentation range for randomly segmenting the sample images based on the relative positional relationship between virtual and real scenes in different virtual shooting tasks; randomly segmenting each sample image into a first part and a second part based on the segmentation range; stitching the first part with the corresponding second part of the sample image after style modification to form a stitched image corresponding to each sample image; using the boundary lines between different parts of each sample image and each stitched image as labels for the corresponding stitched image to form a training dataset; and training a model based on the training dataset to obtain the style transfer model. The style transfer model is trained for different virtual shooting tasks. The segmentation range is determined by a proportional range and an object range. The proportional range is determined around the basic proportion between the first part and the second part. The basic proportion is determined by the ratio between the screen size and the area occupied by the real scene built in front of the screen. The proportional range is determined by the camera's shooting angle range. The object range is set for active objects in the real scene.

2. The method according to claim 1, characterized in that, Style transfer processing is performed on the virtual or real scene regions in the image to be processed using a style transfer model to obtain the processed image, including: The style transfer model is used to transfer the image style of the real scene area in the image to the virtual scene area in the image to be processed, resulting in the processed image.

3. The method according to claim 1, characterized in that, The relative positional relationship between the first part and the second part in each of the sample images corresponds to the relative positional relationship between the virtual scene and the real scene in the image.

4. The method according to claim 1, characterized in that, The method further includes: For the first part of each of the sample images, a style change is performed using a filter template randomly selected from multiple filter templates.

5. An image processing apparatus, characterized in that, The device, used for virtual shooting, includes: The image acquisition module is used to acquire the image to be processed captured by the camera in virtual shooting. The image to be processed is obtained by the camera capturing content based on the virtual scene displayed on the screen and the real scene set up outside the screen. The real scene refers to the scene set up around the screen by placing objects based on shooting requirements. The image to be processed includes a virtual scene area corresponding to the virtual scene and a real scene area corresponding to the real scene. The image processing module is used to perform style transfer processing on the virtual or real scene areas in the image to be processed using a style transfer model to obtain the processed image, wherein the real scene areas and virtual scene areas in the processed image have the same image style. The device further includes: a boundary determination module, used to identify the style of the image to be processed using a style transfer model, determine the boundary line indicating the change in image style in the image to be processed, and determine the virtual scene area and the real scene area in the image to be processed based on the boundary line. The device further includes: a sample acquisition module for acquiring multiple sample images; a sample segmentation module for determining a segmentation range for randomly segmenting the sample images based on the relative positional relationship between virtual scenes and real scenes in different virtual shooting tasks; and for randomly segmenting each sample image into a first part and a second part based on the segmentation range; an image stitching module for stitching each first part with the corresponding second part of the sample image after style modification to form a stitched image corresponding to each sample image; a dataset generation module for using the boundary lines between different parts of each sample image and each stitched image as labels for the corresponding stitched image to form a training dataset; and a model training module for training a model based on the training dataset to obtain the style transfer model. The style transfer model is trained for different virtual shooting tasks. The segmentation range is determined by a proportional range and an object range. The proportional range is determined around the basic proportion between the first part and the second part. The basic proportion is determined by the ratio between the screen size and the area occupied by the real scene built in front of the screen. The proportional range is determined by the camera's shooting angle range. The object range is set for active objects in the real scene.

6. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 4 when executing instructions stored in the memory.

7. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 4.

8. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, characterized in that, When the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the method according to any one of claims 1 to 4.

Images