Personalized scene image processing method and device

Abstract

The application of the invention is a divisional application of the Chinese application 201810717653. X. The invention discloses a personalized scene image processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a touch event track in a screen region according to a touch event triggered in the screen region of terminal equipment; according to the touch event track, generating a virtual model, and obtaining a model view mapped by the virtual model in a view field range of the terminal equipment; and superposing the model view and a scene image acquired by the terminal equipment to obtain a personalized scene image. According to the method, the customized virtual model can be formed according to the graph customized and drawn by the user, so that the flow for downloading the virtual model and the space for storing the virtual model are saved, the labor cost for designing the virtual model is reduced, and the types of the virtual model are enriched.

Description

[0001] This application is a divisional application of the Chinese application with application number 201810717653.X, application date July 3, 2018, and invention name “Personalized scene image processing method and device”. Technical Field

[0002] The present invention relates to the field of image processing technology, and in particular to a method and device for processing personalized scene images, an electronic device, and a computer-readable storage medium. Background Art

[0003] Augmented Reality (AR) has been one of the research hotspots of many well-known foreign universities and research institutions in recent years. AR technology is an emerging technology developed in recent years. It is a technology that calculates the position and angle of camera images in real time and adds corresponding images, videos, and 3D models. Its core is to integrate virtual content and real content in real time to form an interaction between virtual and reality, thereby creating a new experience.

[0004] Currently, most of the software apps that can capture short videos are based on the capabilities of ARkit (augmented reality tool), which uses animation, pets, etc. as prototypes for complex 3D modeling, and projects virtual models onto a certain plane in the real space, such as a desktop or ground, to merge the captured real scene with the virtual model projected in the space to form a personalized video.

[0005] Obviously, in the prior art, virtual models are all pre-configured, and users can only choose from the pre-configured virtual models, so the types of virtual models available to users are limited. If the types of configured virtual models are increased, the space occupied by the software APP will increase, as well as the traffic cost of downloading the software APP, and the human cost of virtual model design will be further increased. Summary of the invention

[0006] In order to solve the problem in the related art that software APP occupies a large space and has a high download traffic cost, the present invention provides a method for processing personalized scene images.

[0007] In one aspect, the present invention provides a method for processing a personalized scene image, comprising:

[0008] According to a touch event triggered in a screen area of ​​a terminal device, a touch event trajectory in the screen area is acquired, wherein the touch event trajectory is a trajectory formed based on a change in a triggering position of the touch event;

[0009] generating a virtual model according to the touch event trajectory, wherein the virtual model is a graph projected in space by the touch event trajectory;

[0010] Obtaining a model view of the virtual model in the current position of the terminal device;

[0011] The model view is superimposed on the scene image collected by the terminal device to obtain a personalized scene image.

[0012] In another aspect, the present invention provides a device for processing a personalized scene image, the device comprising:

[0013] A trajectory acquisition module, used to acquire a touch event trajectory in the screen area of ​​the terminal device according to a touch event triggered in the screen area of ​​the terminal device, wherein the touch event trajectory is a trajectory formed based on a change in a trigger position of the touch event;

[0014] A model building module, used to generate a virtual model according to the touch event trajectory, wherein the virtual model is a graph projected in space by the touch event trajectory; and obtain a model view of the virtual model in the current position of the terminal device;

[0015] The scene overlay module is used to overlay the model view with the scene image collected by the terminal device to obtain a personalized scene image.

[0016] Furthermore, the present invention also provides an electronic device, the electronic device comprising:

[0017] processor;

[0018] a memory for storing processor-executable instructions;

[0019] Wherein, the processor is configured to execute the above-mentioned personalized scene image processing method.

[0020] Furthermore, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to complete the above-mentioned personalized scene image processing method.

[0021] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0022] The technical solution provided by the present invention projects the touch event trajectory into the spatial coordinate system to construct a virtual model according to the trajectory of the triggered touch event, and then during the movement of the terminal device, the model view and scene image captured in the posture can be superimposed according to the real-time posture of the terminal device to form a personalized scene image. As a result, the virtual model of the present invention is no longer pre-configured, and a customized virtual model can be formed according to the graphics drawn by the user, thereby saving the traffic for downloading the virtual model and the space for storing the virtual model, reducing the labor cost of designing the virtual model, and enriching the types of virtual models.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0025] Figure 1 is a schematic diagram of an implementation environment according to the present invention;

[0026] Figure 2 is a block diagram of a device according to an exemplary embodiment;

[0027] Figure 3 is a flowchart of a method for processing a personalized scene image according to an exemplary embodiment;

[0028] Figure 4 It is an effect diagram of displaying 3D models by existing technology;

[0029] Figure 5 is Figure 3 A flowchart of a method for processing a personalized scene image based on the corresponding embodiment;

[0030] Figure 6 This is a schematic diagram of the display effects of three types of trajectory material styles;

[0031] Figure 7 yes Figure 3 Detailed flowchart of step 310 in the corresponding embodiment;

[0032] Figure 8 is a schematic diagram showing the effect of display interface switching according to an exemplary embodiment of the present invention;

[0033] Fig. 9 is a display effect diagram of brush switching according to an exemplary embodiment of the present invention;

[0034] Fig.10 yes Figure 3 Detailed flowchart of step 330 in the corresponding embodiment;

[0035] Fig.11 yes Figure 3 Detailed flowchart of step 350 in the corresponding embodiment;

[0036] Fig.12 This is a schematic diagram of the ARKit framework architecture;

[0037] Fig.13It is a schematic diagram of the process of implementing the present invention based on the ARkit framework;

[0038] Fig.14 is a block diagram of a device for processing personalized scene images according to an exemplary embodiment;

[0039] Fig.15 yes Fig.14 Detailed block diagram of the trajectory acquisition module in the corresponding embodiment;

[0040] Fig.16 yes Fig.14 Detailed block diagram of the model building module in the corresponding embodiment;

[0041] Fig.17 yes Fig.14 Detailed block diagram of the view reconstruction module in the corresponding embodiment. DETAILED DESCRIPTION

[0042] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0043] Figure 1 1 is a schematic diagram of an implementation environment involved in the present invention. The implementation environment includes: a terminal device 110 with a camera 111, a software client 112 is installed in the terminal device 110, and the terminal device 110 can use the personalized scene image processing method provided by the present invention by running the software client 112, calling the camera to collect the scene image and superimposing a customized virtual model on the scene image.

[0044] Figure 2 is a block diagram of a device 200 according to an exemplary embodiment. For example, the device 200 may be Figure 1 The terminal device 110 in the illustrated implementation environment. The terminal device 110 may be a mobile terminal such as a smart phone, a tablet computer, etc.; the terminal device 110 may also be a smart home device such as a smart camera, etc.

[0045] Reference Figure 2 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , a sensor component 214 , and a communication component 216 .

[0046] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 218 to execute instructions to perform all or part of the steps of the method described below. In addition, the processing component 202 may include one or more modules to facilitate the interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate the interaction between the multimedia component 208 and the processing component 202.

[0047] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The memory 204 also stores one or more modules, which are configured to be executed by the one or more processors 218 to complete the following: Figure 3 , Figure 7 , Fig.10 , Fig.11 All or part of the steps in any of the methods shown.

[0048] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0049] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel. If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. The screen may also include an organic light emitting display (OLED).

[0050] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), which is configured to receive external audio signals when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0051] The sensor assembly 214 includes one or more sensors for providing various aspects of status evaluation for the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of the components, and the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200 and the temperature change of the device 200. In some embodiments, the sensor assembly 214 can also include a magnetic sensor, a pressure sensor or a temperature sensor.

[0052] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi (WIreless-Fidelity). In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth technology and other technologies.

[0053] In an exemplary embodiment, the device 200 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors, digital signal processing devices, programmable logic devices, field programmable gate arrays, controllers, microcontrollers, microprocessors or other electronic components to perform the following method.

[0054] Figure 3 FIG. 1 is a flowchart of a method for processing a personalized scene image according to an exemplary embodiment. The method for processing a personalized scene image can be Figure 1 The execution of the terminal device 110 in the implementation environment shown can also be understood as being executed by the software client 112 running on the terminal device 110, for example, a short video software APP. In the following method embodiments, for the sake of convenience, the execution subject of each step is described as the software client 112, but this does not constitute a limitation. Figure 3 As shown, the method may be executed by the software client 112 and may include the following steps.

[0055] In step 310, according to a touch event triggered in a screen area of ​​a terminal device, a touch event track in the screen area is acquired.

[0056] Among them, the terminal device can be a mobile terminal with an image acquisition function, such as a smart phone or a tablet computer, or other smart devices with an image acquisition function, such as a smart camera. The screen area refers to the area in the display interface of the terminal device where patterns can be drawn by touch. The user can draw patterns with a finger or a stylus in the screen area of ​​the terminal device. A touch event refers to an event generated when a user touches the screen with a finger or a stylus in the screen area. The touch event trajectory is the trajectory of a finger or a stylus touching the screen area, including the trigger position of the touch event. The software client receives the touch event triggered in the screen area of ​​the terminal device, and forms a touch event trajectory as the trigger position of the touch event continuously changes.

[0057] In step 330 , a virtual model is generated according to the projection of the touch event trajectory in the space coordinate system.

[0058] Among them, the spatial coordinate system is an absolute coordinate system, which can be a world coordinate system (with a point in space as the origin). All objects have corresponding position coordinates in the spatial coordinate system, and thus the terminal device also has corresponding position coordinates in the spatial coordinate system. Among them, the projection of the touch event trajectory refers to converting the pixel point position (u, v) in the display interface of the terminal device into the corresponding position in the spatial coordinate system (x, y, z are preset constants) according to the conversion rules between the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system. That is, the pixel point position triggered by the touch event in the screen area of ​​the display interface is mapped to the position point in the spatial coordinate system, so that the touch event trajectory can be mapped to one or more position points in the spatial coordinate system. Generating a virtual model refers to forming a three-dimensional stereoscopic figure at the position point mapped in the spatial coordinate system.

[0059] In step 350, according to the position and posture of the terminal device in the spatial coordinate system, a model view of the virtual model mapped within the field of view of the terminal device is reconstructed.

[0060] It should be noted that the terminal device is equipped with a gyroscope and an accelerometer, which can collect the posture changes of the terminal device in real time. The posture includes the orientation of the terminal device and its position in the spatial coordinate system. According to its real-time position and orientation in the spatial coordinate system, the terminal device maps the position points of the virtual model in the spatial coordinate system (i.e., the world coordinate system) to the pixel positions in the display interface of the terminal device according to the conversion rules between the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. Fill the corresponding pixel values ​​at the pixel positions mapped by the terminal device with the virtual model to obtain the model view. It should be explained that the model view can be regarded as the picture obtained by the terminal device in the current posture by capturing the image of the virtual model through the camera configured by itself, in which the area other than the virtual model can be regarded as transparent.

[0061] In step 370, the model view is superimposed with the scene image captured by the terminal device in the posture to obtain a personalized scene image.

[0062] The scene image refers to the image of the real world captured by the camera configured by the terminal device itself. When the terminal device moves, the model view reconstructed at the current posture is superimposed with the scene image captured by the camera at the current posture. The superposition method can be to use the scene image as the background and overlay the model view on the scene image. In this way, the virtual model can be regarded as an object fixed in the real world. When the image of the real world is captured, the constructed virtual model is also captured. Personalized scene image refers to the superposition of the model view on the real captured scene image.

[0063] It should be noted that in the prior art, virtual models are pre-configured and stored in terminal devices. Downloading and storing a large number of virtual models takes up a large amount of space and download traffic. Moreover, the construction of a large number of virtual models requires a lot of manpower costs. Furthermore, the types of virtual models available for users to choose from are limited, and thus cannot meet the needs of all users. Figure 4 As shown, the prior art uses cartoons and pets as prototypes for 3D modeling. Building a 3D model of a complex object requires a lot of manpower and material resources, and the model is relatively simple, and the interactivity is poor during user use.

[0064] The technical solution provided by the above exemplary embodiment of the present invention projects the touch event trajectory into the spatial coordinate system to construct a virtual model according to the triggered touch event trajectory, and then during the movement of the terminal device, the model view and scene image captured in the posture can be superimposed according to the real-time posture of the terminal device to form a personalized scene image. As a result, the virtual model of the present invention is no longer pre-configured, and a custom virtual model can be formed according to the graphics drawn by the user, thereby saving the traffic for downloading the virtual model and the space for storing the virtual model, reducing the labor cost of designing the virtual model, and enriching the types of virtual models.

[0065] In an exemplary embodiment, Figure 5 As shown, before the above step 310, the method for processing personalized scene images provided by the present invention further includes the following steps:

[0066] In step 301, multiple track material styles are displayed according to the received track drawing request;

[0067] Specifically, the software client displays a virtual model drawing button in the terminal device display interface, receives a trajectory drawing request triggered by the user, and then responds to the virtual model drawing request triggered by the user, obtains multiple trajectory material styles to be selected, and displays the icon corresponding to each trajectory material style. Among them, the trajectory material style is used to indicate the brush style of the virtual model drawn by the user through touch.

[0068] In step 302, a target trajectory style is obtained according to the selection of multiple trajectory material style triggers; and the virtual model is configured according to the target trajectory style.

[0069] The user can trigger and select one of the multiple track material styles displayed. The target track style refers to the style selected by the user from the multiple track material styles. After displaying multiple track material style icons, the software client receives a trigger instruction from the user clicking one of the icons. The software client obtains the target track style selected by the user according to the user's selection of the multiple track material styles triggered by the trigger instruction. Then, the subsequent virtual model can be generated according to the brush style configured by the target track style. Figure 6 The following is a schematic diagram of the effects of three types of track materials.

[0070] In an exemplary embodiment, Figure 7 As shown, the above step 310 specifically includes:

[0071] In step 311, the image acquisition device of the terminal device is started to acquire the current scene image;

[0072] Specifically, the image acquisition device refers to the camera configured by the terminal device itself. The current scene image refers to the image of the real scene acquired by the camera of the terminal device at the current moment. After receiving the trigger instruction of the user triggering the selection of the target trajectory style, the software client calls the camera of the terminal device to acquire the image of the real scene.

[0073] In step 312, a touch event triggered in the current scene image display interface is received, and a triggering position of the touch event is recorded, where the triggering position of the touch event constitutes the touch event track.

[0074] The software client displays the current scene image in the display interface of the terminal device, receives the touch event triggered in the current scene image display interface, and records the trigger position of the touch event. The connection line of one or more trigger positions constitutes the touch event trajectory.

[0075] Figure 8 It is a schematic diagram of the effect of display interface switching shown in an exemplary embodiment of the present invention.

[0076] like Figure 8 As shown, the terminal device opens the software client, and the software client receives the instruction of the user clicking the "shoot" button and enters the front camera shooting interface 810.

[0077] The software client receives the instruction of the user clicking the "magic" button 801, and displays the operation panel 802 corresponding to the "magic" button on the display interface. The software client receives the instruction of the user clicking the "magic brush" button 803 in the operation panel 802, and enters the magic brush selection interface 820. The magic brush selection interface 820 displays a variety of track material styles (i.e., brush types), such as "a lot of money", "smoke", "colorful brush" and other brush types. The user can select any brush type, and the user is shown a prompt of "drawing in three-dimensional space" to guide the user to create.

[0078] After receiving the instruction from the user to select any type of brush, the software client can receive the touch event triggered on the screen, obtain the touch event trajectory, and project the touch event trajectory in space to generate a three-dimensional image. The three-dimensional image is fixed based on space. The user adjusts the position and angle of the terminal device camera to display the three-dimensional image at different angles in the terminal device display interface (see interfaces 830 and 840).

[0079] In one embodiment, if Fig. 9 As shown, the user can freely draw in the space with the collected scene image, see interface 910. Since multiple brushes cannot be used at the same time, when the user switches the brush, the user can be prompted to clear the currently created track pattern, see interface 920.

[0080] Based on the above exemplary embodiments, Fig.10 As shown, the above step 330 specifically includes:

[0081] In step 331, according to the position of the terminal device in the spatial coordinate system, the touch event trajectory is projected onto a plane at a preset distance from the current position in the field of view direction of the terminal device;

[0082] The touch event occurs when the user's finger or stylus touches the screen. The field of view direction of the terminal device refers to the direction in which the camera configured by the terminal device itself acquires images. The pose includes position and attitude data. The position is used to represent the position of the camera in the terminal device in the spatial coordinate system. The attitude is used to represent the acquisition direction of the camera in the terminal device.

[0083] Specifically, when a touch event occurs, the software client converts the pixel position coordinates where the touch event occurs in the display interface from the pixel coordinate system to the position point in the world coordinate system according to the position (including position and posture) of the camera in the terminal device in the spatial coordinate system (finding the corresponding point in the world coordinate system from a specified point in the pixel coordinate system belongs to the prior art and will not be repeated here). It should be explained that according to the conversion rules between the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system, a coordinate point in the world coordinate system can find a corresponding pixel point in the image, but conversely, a depth value is missing when finding the corresponding point in the world coordinate system through a pixel point in the image. Therefore, the present invention projects the touch event trajectory into a plane at a preset distance from the current position in the spatial coordinate system by presetting a depth value (i.e., a preset distance), and according to the posture of the camera, the plane needs to be located in the acquisition direction of the camera of the terminal device. Therefore, according to the pixel coordinate position of the touch event trajectory in the pixel coordinate system, the corresponding position point in the world coordinate system can be found.

[0084] In step 332, a virtual model is generated at the projection position of the touch event trajectory on the plane according to the configured target trajectory pattern.

[0085] Specifically, the plane refers to a plane at a preset distance from the camera and in the direction of camera image acquisition in the spatial coordinate system. The software client projects the acquired touch event trajectory onto the above plane in the spatial coordinate system, and obtains the projection position point of the touch event trajectory in the pixel coordinate system within the above plane in the spatial coordinate system through the conversion rule between the pixel coordinate system and the world coordinate system. In one embodiment, the touch event trajectory includes a plurality of pixel point position coordinates, and thus the projection position includes a plurality of position points in space corresponding to the pixel point position coordinates of the touch event trajectory.

[0086] The target track style is one of the styles selected by the user from a variety of track material styles, that is, the type of brush. The software client forms a three-dimensional figure indicated by the target track style at the projection position of the touch event track in the spatial coordinate system according to the target track style selected by the user. The three-dimensional figure generated at the projection position can be considered as a virtual model. The virtual model can be considered fixed in space. When the terminal device adjusts its posture, it can capture images of the virtual model at different angles.

[0087] In an exemplary embodiment, Fig.11 As shown, the above step 350 specifically includes:

[0088] In step 351, the field of view of the terminal device in the spatial coordinate system is located according to the position of the terminal device in the spatial coordinate system;

[0089] It should be noted that as the terminal device moves, the gyroscope and accelerometer configured in the terminal device itself can collect the position and posture of the terminal device in the spatial coordinate system in real time, and then obtain the position and posture of the camera configured in the terminal device itself in the spatial coordinate system. Positioning the field of view of the terminal device in the spatial coordinate system means re-establishing the camera coordinate system based on the position and shooting direction of the camera. The origin of the camera coordinate system is the optical center of the camera, the x-axis and y-axis are parallel to the X and Y axes of the image, and the z-axis is the optical axis of the camera, which is perpendicular to the image plane (i.e., the screen display interface). The intersection of the optical axis and the image plane is the origin of the image coordinate system, and the rectangular coordinate system formed is the camera coordinate system. The origin of the pixel coordinate system is the upper left corner of the image (screen display interface).

[0090] In step 352, the virtual model within the field of view is mapped to the display area of ​​the terminal device to form a model view.

[0091] Specifically, according to the conversion rules between the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system, the spatial position points of the constructed virtual model in the world coordinate system are converted into the pixel position coordinates in the pixel coordinate system. Since the display area of ​​the terminal device is limited, according to the pixel position coordinates obtained by the conversion, the pixel values ​​consistent with the corresponding position points of the virtual model are filled in the corresponding pixel position coordinates in the display area of ​​the terminal device, thereby obtaining a pattern mapped by the virtual model in the display area of ​​the terminal device, which is the model view.

[0092] In an exemplary embodiment, the above step 370 specifically includes:

[0093] The scene image captured by the terminal device in the pose is used as the background, and the model view is overlaid on the scene image to obtain a personalized scene image.

[0094] Specifically, for a scene image captured by a terminal device in a certain posture and a model view reconstructed in the posture, the software client uses the scene image as a background and overlays a pattern mapped by the virtual model on the scene image to obtain a personalized scene image.

[0095] In an exemplary embodiment, the above step 370 specifically includes:

[0096] The model view is superimposed frame by frame onto the image sequence captured by the terminal device in the given position to obtain a personalized scene video.

[0097] It should be noted that the video recording function can be started to collect image sequences. While collecting image sequences, for the posture of the terminal device when collecting each frame of the image, the model view in the posture is superimposed on the collected image, and the model view in the current posture is superimposed on the collected scene image frame by frame, so that a personalized scene video can be obtained.

[0098] In one embodiment, the personalized scene image processing method provided by the present invention can be implemented based on the ARKit framework. ARKit provides two virtual augmented reality views, including a scene image with a 3D effect and a scene image with a 2D effect. Fig.12 This is a schematic diagram of the ARKit framework architecture. Fig.12 As shown, the role of UIView (user interface view) is to display the view in the display interface of the terminal device, and the role of ARSCNView (AR view) is also to display a 3D scene, but this 3D scene is composed of real-world images captured by the camera. ARSCNView is just a view container, and its role is to manage an ARSession (AR session). In a complete virtual augmented reality experience, the ARKit framework is only responsible for transforming the real-world image into a 3D scene. This transformation process is mainly divided into two steps: ARCamera (AR camera) is responsible for capturing the camera image as the background view, ARSession is responsible for building the 3D scene, and finally ARSCNView displays the 3D scene.

[0099] Among them, ARSession serves as a communication bridge between ARCamera and ARSCNView. The main participants in building the communication bridge are two ARSessionConfiguration (session configuration) and ARFrame (AR architecture). The main purpose of ARSessionConfiguration is to determine the position and direction of the camera in the 3D world and capture some feature scenes (such as plane capture), that is, to detect the surface of the real world seen by the device camera. It mainly detects the movement, rotation and even rolling of the terminal device by calling the sensors equipped with the terminal device. ARFrame contains relevant information such as the pictures captured by the camera and the camera position. Among them, the node (SCNNode) obtains a view of the virtual model drawn by the user, and the scene class (SCNScene class) superimposes the view with the image taken by the camera to form a personalized scene image.

[0100] Fig.13 This is a schematic diagram of the process of implementing the present invention based on the ARkit framework. Fig.13As shown, in step 1301, the software client receives a user triggering a "magic brush" tag and displays a panel corresponding to the "magic brush" tag. The panel displays a variety of brush styles.

[0101] In step 1302, the user may select one of the multiple brush styles displayed, and the software client receives the user's request to select a specified brush style in the panel.

[0102] In step 1303, the software client responds to the request to run ARSCNView to load the scene SCNScene.

[0103] In step 1304, SCNScene starts the camera ARCamera to start capturing scene images.

[0104] In step 1305, the software client runs ARSCNView and passes the captured scene data to ARsession.

[0105] In step 1306, the software client runs ARsession, which determines the camera's position and posture in real time by managing ARsessionConfiguration and returns ARFame (including scene image, position and posture information, etc.).

[0106] In step 1307, the touch event track drawn by the user is obtained and projected in space to form a virtual model. The ARsessionConfiguration determines the position of the camera in real time in order to determine the real position of the virtual model relative to the camera in space.

[0107] In step 1308, the software client runs SCNNode to capture the view of the virtual model in the current pose.

[0108] In step 1309, the software client runs SCNScene to superimpose the view with the image taken by the camera in the current posture to form a personalized scene image, and displays the personalized scene image on the terminal device through UIView.

[0109] Among them, if the user chooses to switch the brush type, a pop-up window will be displayed to remind the user whether he is clear about the current effect.

[0110] The following is an embodiment of the device of the present invention, which can be used to execute the embodiment of the personalized scene image processing method executed by the above-mentioned software client 110 of the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the embodiment of the personalized scene image processing method of the present invention.

[0111] Fig.14is a block diagram of a personalized scene image processing device according to an exemplary embodiment. The personalized scene image processing device can be used for Figure 1 In the terminal device 110 of the implementation environment shown in the figure, execute Figure 3 , Figure 7 , Fig.10 , Fig.11 All or part of the steps of any of the methods for processing personalized scene images shown. Fig.14 As shown, the apparatus includes but is not limited to: a trajectory acquisition module 1410 , a model construction module 1430 , a view reconstruction module 1450 and a scene overlay module 1470 .

[0112] The track acquisition module 1410 is used to acquire the touch event track in the screen area according to the touch event triggered in the screen area of ​​the terminal device;

[0113] A model building module 1430, configured to generate a virtual model according to the projection of the touch event trajectory in the space coordinate system;

[0114] A view reconstruction module 1450, configured to reconstruct a model view of the virtual model mapped within the field of view of the terminal device according to the position of the terminal device in the spatial coordinate system;

[0115] The scene overlay module 1470 is used to overlay the model view with the scene image captured by the terminal device in the posture to obtain a personalized scene image.

[0116] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned personalized scene image processing method, and will not be repeated here.

[0117] The trajectory acquisition module 1410 may be, for example, Figure 2 A physical structure sensor component 214 in the.

[0118] The model building module 1430, the view reconstruction module 1450, and the scene overlay module 1470 may also be functional modules for executing the corresponding steps in the above-mentioned personalized scene image processing method. It is understood that these modules can be implemented by hardware, software, or a combination of the two. When implemented in hardware, these modules can be implemented as one or more hardware modules, such as one or more application-specific integrated circuits. When implemented in software, these modules can be implemented as one or more computer programs executed on one or more processors, such as Figure 2 The program stored in the memory 204 is executed by the processor 218 of the computer.

[0119] In an exemplary embodiment, the above-mentioned personalized scene image processing device further includes:

[0120] The style acquisition module is used to display multiple track material styles according to the received track drawing request;

[0121] The style selection module is used to obtain a selected target trajectory style according to the selection triggered by the multiple trajectory material styles; the virtual model is configured according to the target trajectory style.

[0122] In an exemplary embodiment, Fig.15 As shown, the trajectory acquisition module 1410 includes:

[0123] The image acquisition unit 1411 is used to start the image acquisition device of the terminal device to acquire the current scene image;

[0124] The event receiving unit 1412 is used to receive a touch event triggered in the current scene image display interface, and record a triggering position of the touch event, where the triggering position of the touch event constitutes the touch event track.

[0125] In an exemplary embodiment, Fig.16 As shown, the model building module 1430 includes:

[0126] A trajectory projection unit 1431 is used to project the touch event trajectory onto a plane at a preset distance from the current position in the field of view direction of the terminal device according to the position of the terminal device in the spatial coordinate system;

[0127] The model generating unit 1432 is configured to generate a virtual model at a projection position of the touch event trajectory on the plane according to a configured target trajectory pattern.

[0128] In an exemplary embodiment, Fig.17 As shown, the view reconstruction module 1450 includes:

[0129] A field of view positioning unit 1451, configured to position the field of view range of the terminal device in the spatial coordinate system according to the position of the terminal device in the spatial coordinate system;

[0130] The model mapping unit 1452 is used to map the virtual model within the field of view to the display area of ​​the terminal device to form a model view.

[0131] In an exemplary embodiment, the scene overlay module 1470 includes:

[0132] A view overlay unit is used to use the scene image captured by the terminal device in the posture as a background, overlay the model view on the scene image, and obtain the personalized scene image.

[0133] In an exemplary embodiment, the scene overlay module 1470 includes:

[0134] The model view is superimposed frame by frame on the image sequence captured by the terminal device in the posture to obtain a personalized scene video.

[0135] Optionally, the present invention further provides an electronic device, which can be used for Figure 1 In the terminal device 110 of the implementation environment shown in the figure, execute Figure 3 , Figure 7 , Fig.10 , Fig.11 All or part of the steps of any of the methods for processing personalized scene images shown. The electronic device comprises:

[0136] processor;

[0137] a memory for storing processor-executable instructions;

[0138] Wherein, the processor is configured to execute the method for processing personalized scene images described in the above exemplary embodiment.

[0139] The specific manner in which the processor of the electronic device in this embodiment performs operations has been described in detail in the embodiment of the method for processing the personalized scene image, and will not be elaborated here.

[0140] In an exemplary embodiment, a storage medium is also provided, which is a computer-readable storage medium, for example, a temporary and non-temporary computer-readable storage medium including instructions. The storage medium, for example, includes a memory 204 of instructions, and the storage medium stores a computer program, which can be executed by the processor 218 of the device 200 to complete the above-mentioned personalized scene image processing method.

[0141] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be performed without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for processing personalized scene images, characterized in that: include: According to a touch event triggered in a screen area of ​​a terminal device, a touch event trajectory in the screen area is acquired, wherein the touch event trajectory is a trajectory formed based on a change in a triggering position of the touch event; generating a virtual model according to the touch event trajectory, wherein the virtual model is a graphic projected in space by the touch event trajectory; Obtaining a model view of the virtual model in the current position of the terminal device; The model view is superimposed on the scene image collected by the terminal device to obtain a personalized scene image.

2. The method according to claim 1, characterized in that The generating a virtual model according to the touch event trajectory includes: A three-dimensional stereoscopic graphic is generated according to the projection of the touch event trajectory in space, and the three-dimensional stereoscopic graphic serves as the virtual model.

3. The method according to claim 2, characterized in that The three-dimensional stereoscopic figure is fixed based on the space.

4. The method according to claim 1, characterized in that: The generating a virtual model according to the touch event trajectory includes: The virtual model is generated according to the projection of the touch event trajectory in the space coordinate system.

5. The method according to claim 4, characterized in that The generating the virtual model according to the projection of the touch event trajectory in the space coordinate system includes: Based on the posture of the terminal device, the touch event trajectory is projected into a plane at a preset distance from the current position in the spatial coordinate system to generate the virtual model. The plane refers to a plane at a preset distance from the camera of the terminal device and in the direction of image acquisition of the camera.

6. The method according to any one of claims 1 to 5, characterized in that: The obtaining of a model view of the virtual model in the current posture of the terminal device includes: Based on the current position of the terminal device, a picture obtained by collecting images of the virtual model by a camera of the terminal device is determined as the model view.

7. The method according to any one of claims 1 to 5, characterized in that: Before acquiring the touch event track in the screen area according to the touch event triggered in the screen area of ​​the terminal device, the method further includes: Display multiple track material styles according to the received track drawing request; According to the selection of the plurality of trajectory material style triggers, a selected target trajectory style is obtained; and the virtual model is configured according to the target trajectory style.

8. The method according to any one of claims 1 to 5, characterized in that: The acquiring the touch event track in the screen area according to the touch event triggered in the screen area of ​​the terminal device includes: Starting the image acquisition device of the terminal device to acquire the current scene image; A touch event triggered in the current scene image display interface is received, and a triggering position of the touch event is recorded, wherein the triggering position of the touch event constitutes the touch event track.

9. The method according to any one of claims 1 to 5, characterized in that: The generating a virtual model according to the touch event trajectory includes: According to the position and posture of the terminal device in the spatial coordinate system, projecting the touch event trajectory onto a plane at a preset distance from the current position in the field of view direction of the terminal device; According to the configured target track pattern, the touch event track is used to generate the virtual model at the projection position of the plane.

10. The method according to any one of claims 1 to 5, characterized in that: The obtaining of the model view of the virtual model in the current posture of the terminal device comprises: Reconstructing the field of view of the terminal device in the spatial coordinate system according to the current position of the terminal device in the spatial coordinate system; The virtual model within the field of view is mapped to the display area of ​​the terminal device to obtain the model view of the virtual model in the current position of the terminal device.

11. The method according to any one of claims 1 to 5, characterized in that: The step of superimposing the model view with the scene image collected by the terminal device to obtain a personalized scene image includes: The scene image collected by the terminal device is used as a background, and the model view is overlaid on the scene image to obtain the personalized scene image.

12. The method according to claim 11, characterized in that The step of using the scene image collected by the terminal device as a background and overlaying the model view on the scene image to obtain the personalized scene image includes: The scene image captured by the terminal device in the current posture is used as the background, and the model view is overlaid on the scene image to obtain the personalized scene image.

13. The method according to any one of claims 1 to 5, characterized in that: The step of superimposing the model view with the scene image collected by the terminal device to obtain a personalized scene image includes: The model view is superimposed frame by frame on the image sequence captured by the terminal device to obtain a personalized scene video.

14. A device for processing personalized scene images, characterized in that: The device comprises: A trajectory acquisition module, used to acquire a touch event trajectory in the screen area of ​​the terminal device according to a touch event triggered in the screen area of ​​the terminal device, wherein the touch event trajectory is a trajectory formed based on a change in a trigger position of the touch event; A model building module, used to generate a virtual model according to the touch event trajectory, wherein the virtual model is a graph projected in space by the touch event trajectory; and obtain a model view of the virtual model in the current position of the terminal device; The scene overlay module is used to overlay the model view with the scene image collected by the terminal device to obtain a personalized scene image.

15. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the personalized scene image processing method described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to complete the method for processing personalized scene images described in any one of claims 1-13.

Images