Image processing method and related apparatus

Abstract

This application provides an image processing method and related apparatus, relating to the field of terminal technology. The method includes adding motion effects to an image displayed on an electronic device when taking a photo or recording a video, thereby enhancing the visual effect of the photograph or video.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method and related apparatus. Background Technology

[0002] In daily life, people often use electronic devices to record videos. However, the visual quality of the recorded videos is currently quite poor. Summary of the Invention

[0003] This application provides an image processing method and related apparatus, applicable to the field of terminal technology. It helps improve the visual effect of images displayed on electronic devices.

[0004] In a first aspect, embodiments of this application propose an image processing method, including:

[0005] A first interface is displayed, which shows an image captured by the camera of the electronic device. The first interface includes a first button, and the image displayed on the first interface does not contain any target animation effects, which may include particle rendering and / or animation rendering. In response to an operation on the first button, a second interface is displayed, which shows an image captured by the camera of the electronic device, and the image displayed on the second interface contains target animation effects.

[0006] For example, the first interface can be, for instance, Figure 5 The interface shown in (b) is shown in the diagram. The first button could be, for example, […]. Figure 5 The rendering mode button is shown in (b) above. Particle rendering can be referenced... Figure 5 The rendering shown in (c) can be referenced for animation rendering. Figure 5 The rendering shown in (c) is an example. Again, by way of example, the first interface could be, for example, Figure 6 In the interface shown in (b), the first button can be, for example, Figure 6 The rendering control buttons are shown in (b) above. As another example, the first interface could be, for example, a... Figure 6 In the interface shown in (a), the first button can be, for example, Figure 6 (a) Portrait Mode button.

[0007] The second interface can be, for example, Figure 5 The interface shown in (c) can also be Figure 6 The interface shown in (d) is shown in the image.

[0008] This application embodiment displays a first interface including a first button, and the image displayed on the first interface does not have a target animation effect. In response to an operation on the first button, a second interface is displayed. The second interface is used to display an image captured by the camera of the electronic device, and the image displayed on the second interface has a target animation effect added. This allows the target animation effect to be added to the displayed image, thereby improving the visual effect of the image displayed by the electronic device.

[0009] In one possible implementation, the second interface includes a second button, and the method further includes:

[0010] In response to an action on the second button in the second interface, the third interface is displayed, and the image displayed in the third interface does not have the target animation effect added; in response to an action on the second button in the third interface, the fourth interface is displayed, and the image displayed in the fourth interface has the target animation effect added.

[0011] For example, the third interface can be, for instance, Figure 6 In the interface shown in (f), the second button could be, for example, Figure 6 The rendering control button shown in (f) in the fourth interface can be, for example, the fourth interface. Figure 6 The interface shown in (d) is shown in the image.

[0012] In this embodiment of the application, the activation or deactivation of the target animation effect is controlled by a second button, which can improve the convenience of controlling the activation or deactivation of the target animation effect.

[0013] In one possible implementation, the method further includes, before displaying the first interface:

[0014] The fifth screen is displayed, which includes the camera app icon; the first screen is also displayed, including:

[0015] In response to an action on the camera app icon, the first screen is displayed, which is the camera preview screen.

[0016] For example, the fifth interface could be, for instance, Figure 5 The interface shown in (a) is, for example, the first interface. Figure 5 The interface shown in (b) allows the target animation to be hidden when entering the camera preview interface, and then enabled when the user needs it, thus preserving the authenticity of the captured image and improving the user experience.

[0017] In one possible implementation, the second interface is a video recording interface, which includes a third button. The method also includes:

[0018] In response to the first operation on the third button, video recording is started; in response to the second operation on the third button, video recording ends and the target video is obtained. The target video has a target animation effect added. The target video includes the video recorded by the camera within the target time period, which includes the time period between the response to the first operation and the response to the second operation.

[0019] The target video can be a video obtained through video recording. In this embodiment, after the target animation effect is enabled, the recorded video also has the target animation effect added, which can improve the visual effect of the recorded video.

[0020] In one possible implementation, before displaying the second interface in response to an action on the first button, the method further includes:

[0021] The method further includes: acquiring a first image captured by the camera of an electronic device; extracting information from the first image, the information of which includes scene information and / or object information, the scene information representing the scene of the first image and the object information representing the action of the first object in the first image; acquiring a first mapping relationship, the first mapping relationship representing the mapping relationship between image information and motion effects; and acquiring the target motion effect corresponding to the information of the first image from the first mapping relationship; the method also includes: when displaying a second interface, displaying the first image on the second interface and adding the target motion effect corresponding to the information of the first image to the first image.

[0022] This application embodiment acquires a first image captured by the camera of an electronic device; extracts information from the first image, the information of the first image including scene information and / or object information of the first image, the scene information representing the scene of the first image, and the object information representing the action of the first object in the first image; acquires a first mapping relationship, the first mapping relationship representing the mapping relationship between image information and motion effects; and acquires the target motion effect corresponding to the information of the first image from the first mapping relationship; the method further includes: when displaying a second interface, displaying the first image on the second interface, and adding the target motion effect corresponding to the information of the first image to the first image, so that the corresponding motion effect can be selected according to the information of the image, which can improve the adaptability of adding the target motion effect and improve the user experience.

[0023] In one possible implementation, the method further includes the following steps before displaying the second interface:

[0024] The depth information of the first image is obtained, including the first depth of any image region in the first image, wherein the first depth of the image region indicates the depth of the image region in the first image; the second depth of the target animation is obtained, wherein the second depth indicates the depth of the target animation in the first image; and the target animation corresponding to the information of the first image is added to at least one image region based on the first depth and the second depth of any image region, wherein for the image region to which the target animation is added, the first depth of the image region is greater than the second depth of the target animation added to the image region.

[0025] In this context, the image region can also be simply referred to as a region. In this embodiment, a single pixel can be considered as an image region, or multiple adjacent pixels can be considered as an image region; no limitation is imposed. This embodiment can be referred to in the relevant descriptions of S808-S814, which will not be repeated here.

[0026] In this embodiment, the target motion effect is added to an image region with a first depth greater than the second depth of the target motion effect, while the target motion effect is not added to an image region with a first depth less than or equal to the second depth of the target motion effect. This reduces resource waste when adding motion effects and improves resource utilization.

[0027] In one possible implementation, the depth information of the first image includes L-level depth information, the first depth of an image region in the K-level depth information is related to the first depths of at least two image regions in the (K-1)-level depth information, an image region in the K-level depth information includes at least two image regions in the (K-1)-level depth information, 1 < K ≤ L, and K and L are integers. Based on the first and second depths of any image region, a target animation corresponding to the information of the first image is added to at least one image region, including:

[0028] Obtain the first depth of the image region where the target motion effect is located from the K-th level depth information; compare the first depth obtained from the K-th level depth information with the second depth; if the first depth obtained from the K-th level depth information is greater than the second depth, add the target motion effect to the image region where the target motion effect is located in the K-th level depth information; if the first depth obtained from the K-th level depth information is less than or equal to the second depth, obtain the first depth of the image region where the target motion effect is located from the (K-1)-th level depth information, so that if the first depth obtained from the (K-1)-th level depth information is greater than the second depth, add the target motion effect to the image region where the target motion effect is located in the (K-1)-th level depth information.

[0029] The specific values ​​of L and K can be set as needed and are not restricted here. For example, L can be 4.

[0030] In this embodiment, the depth information of the first image includes L-level depth information. The first depth of one image region in the K-level depth information is related to the first depth of at least two image regions in the (K-1)-level depth information. One image region in the K-level depth information includes at least two image regions in the (K-1)-level depth information. Then, the first depth of the image region where the target animation is located is obtained from the K-level depth information. The size of the first depth obtained from the K-level depth information is compared with the second depth. If the first depth obtained from the K-level depth information is greater than the second depth, the target animation is added to the image region where the target animation is located in the K-level depth information. If the first depth obtained from the K-level depth information is less than or equal to the second depth, the first depth of the image region where the target animation is located is obtained from the (K-1)-level depth information. In this way, if the first depth obtained from the (K-1)-level depth information is greater than the second depth, the target animation is added to the image region where the target animation is located in the (K-1)-level depth information. This can save the computing resources required for depth comparison.

[0031] In one possible implementation, the first depth of an image region in the K-th level depth information is the minimum of the first depths of at least two image regions in the (K-1)-th level depth information.

[0032] Secondly, embodiments of this application provide a flowchart of another image processing method, which may include:

[0033] The first interface displays an image captured by the electronic device's camera. This first interface includes a fourth and a fifth button, and the image displayed on the first interface does not include particle rendering or animation. In response to an action on the fourth button in the first interface, a sixth interface is displayed. This sixth interface displays an image captured by the electronic device's camera, includes a fifth button, and the image displayed on the sixth interface does not include particle rendering but does include animation. In response to an action on the fifth button in the sixth interface, a seventh interface is displayed. This seventh interface displays an image captured by the electronic device's camera, and the image displayed on the seventh interface includes both particle rendering and animation.

[0034] For example, the sixth interface could be, for instance, Figure 7 The interface shown in (c) is as follows. The fourth button could be, for example, an animation control button. The fifth button could be, for example, a particle control button. For example, the sixth interface could be, for example,... Figure 7 The interface shown in (e) is shown in the image.

[0035] In this embodiment, a first interface is displayed, which displays images captured by the camera of the electronic device. The first interface includes a fourth button and a fifth button. The images displayed on the first interface do not have particle rendering or animation rendering. In response to an operation on the fourth button on the first interface, a sixth interface is displayed, which displays images captured by the camera of the electronic device. The sixth interface includes a fifth button. The images displayed on the sixth interface do not have particle rendering but have animation rendering. In response to an operation on the fifth button on the sixth interface, a seventh interface is displayed, which displays images captured by the camera of the electronic device. The images displayed on the seventh interface have particle rendering and animation rendering. That is, particle rendering and animation rendering are controlled to be turned on by different buttons, which can improve the flexibility of the activated animation effects.

[0036] In one possible implementation, the seventh interface includes a fourth button and a fifth button. After displaying the seventh interface in response to an action on the fifth button in the sixth interface, the method further includes:

[0037] In response to an action on the fourth button in the seventh interface, an eighth interface is displayed. The eighth interface displays an image captured by the camera of the electronic device. The eighth interface includes a fifth button. The image displayed in the eighth interface has particle rendering added but no animation rendering added.

[0038] In response to an action on the fifth button in the eighth interface, the ninth interface is displayed. The ninth interface displays an image captured by the camera of the electronic device. The image displayed in the ninth interface does not have particle rendering or animation rendering.

[0039] For example, the eighth interface could be, for instance, Figure 7 The interface shown in (g) is, for example, the ninth interface. Figure 7 The interface shown in (i) is shown in the diagram.

[0040] In this embodiment, the particle rendering switch is controlled by the same button, and the animation rendering switch is controlled by another button, which facilitates the control of both particle rendering and animation rendering, and improves the user experience.

[0041] Thirdly, embodiments of this application provide an image processing apparatus, which may be an electronic device, a chip, or a chip system within an electronic device. The image processing apparatus may include a display unit and a processing unit. When the image processing apparatus is an electronic device, the display unit may be a display screen. The display unit is used to perform display steps to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is an electronic device, the processing unit may be a processor. The image processing apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is a chip or a chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0042] Fourthly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory is used to store code instructions, and the processor is used to execute the code instructions to perform the methods described in the first aspect, the second aspect, any possible implementation of the first aspect, or any possible implementation of the second aspect.

[0043] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect, the second aspect, any possible implementation of the first aspect, or any possible implementation of the second aspect.

[0044] Sixthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect, the second aspect, any possible implementation of the first aspect, or any possible implementation of the second aspect.

[0045] Seventhly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect, the second aspect, any possible implementation of the first aspect, or any possible implementation of the second aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0046] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0047] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of a scenario in related technologies;

[0049] Figure 2 A schematic diagram illustrating an image processing method provided in an embodiment of this application;

[0050] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0051] Figure 4 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0052] Figures 5-7 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0053] Figure 8 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0054] Figure 9 This is a schematic diagram of a multi-level depth information provided in an embodiment of this application;

[0055] Figure 10 A schematic diagram illustrating the comparison of multi-level deep compression information provided in an embodiment of this application;

[0056] Figure 11 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0057] Figure 12 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0058] Figure 13 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0059] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0060] 1. Particle rendering

[0061] Particle rendering is a technique that uses tiny particles to simulate dynamic visual effects. These particles can be natural phenomena such as flames, smoke, water droplets, and snowflakes, or special effects such as fragments and sparks. Particle systems create dynamic and realistic visual effects by simulating the behavior and properties of each particle (such as position, velocity, size, and color) and updating and rendering these particles over time. The particle rendering process typically involves particle generation, particle attribute setting, particle updating, and particle rendering. Particle generation can be the creation of particles based on needs. These particles can be generated according to certain rules or patterns, such as being emitted from a point or area, or distributed throughout the scene according to a certain algorithm. Particle attribute setting involves setting initial attributes for each particle, such as color, size, and velocity. These attributes can be adjusted as needed to achieve different visual effects. Particle updating involves updating particle attributes according to physical laws or custom behavioral rules. For example, particles may be affected by gravity, wind, or other forces, thus changing their position and velocity. Particle rendering involves using graphics rendering techniques to draw particles onto a display screen. This typically involves calculating the position of each particle on the display screen and drawing it as a point, circle, or other shape. During the rendering process, factors such as lighting, shadows, and transparency also need to be considered to achieve realistic visual effects. It should be noted that particles can be 2D or 3D particles.

[0062] 2. Animation rendering

[0063] In video rendering, animation rendering is applied to various scenarios, such as generating animated text during speeches. Animated text rendering refers to transforming static text into a visually dynamic presentation to enhance the visual appeal and information delivery of a speech. Animation rendering can include frame animation rendering, skeletal animation rendering, and transform animation. Frame animation is a technique that creates a sense of motion by continuously playing a series of static images (frames). In frame animation, each frame is an independent image, and these images are played rapidly and continuously in a certain order, thus forming the animation. Skeletal animation is a technique in computer graphics that simulates the movement of skeletal structures. It drives the animation of the entire model by defining bones and joints in the model and controlling the movement of the bones. Skeletal animation technology can achieve character animation and biological motion simulation, making characters or objects appear to move more naturally and smoothly in animation. Transform animation refers to using the transform property of cascading style sheets (CSS) to change the shape, size, and position of elements, and to create various animation effects. The transform property can accept various parameters, such as rotation, scaling, and skeletal, to create a variety of complex visual effects.

[0064] 3. Other terms

[0065] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0066] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0067] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0068] 4. Electronic equipment

[0069] The electronic devices in this application embodiment may include handheld devices with shooting functions, vehicle-mounted devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving vehicles, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, in-vehicle devices, wearable devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0070] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0071] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0072] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0073] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0074] The following examples illustrate the scenarios described in this application.

[0075] In some example situations, such as Figure 1 As shown, the electronic device can record video, and during the recording process, the recording window in the recording interface displayed on the electronic device shows the video footage being recorded.

[0076] like Figure 1 The interface shown in (a) includes a preview window 201 and a shutter button 203. The preview window 201 can display images captured by the electronic device, including a first person and a second person. Then, when the electronic device detects user operation on the shutter button 203, the electronic device starts recording video. During the video recording process, the image displayed in the preview window 201 may change, such as... Figure 1 As shown in (b), preview window 201 includes the first person but excludes the second person. Then, when the electronic device is recording video, as... Figure 1 As shown in (c), when the electronic device detects a user operation on the shutter button 203, it can generate a video, which may include... Figure 1 (a)- Figure 1 (b) The screen displayed in preview window 201 during this time period.

[0077] Generally, the video displayed on electronic devices is the actual captured footage without any rendering effects. This can result in poor visual quality in the recorded video.

[0078] In view of this, embodiments of this application propose an image processing method and related apparatus that can improve the visual effect of images displayed by electronic devices.

[0079] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating an image processing method provided in an embodiment of this application. Figure 2 As shown, electronic devices can use artificial intelligence (AI) to reason and analyze image-related information such as scene information and character information, and then call the particle system to perform particle rendering and the animation system to perform animation rendering according to the scene information and character information.

[0080] For example, scene information may include, but is not limited to, location information and weather information. Location information may include, but is not limited to, space, ocean, and fireworks. Weather information may include, but is not limited to, rain, snow, and lightning. Character information may include, but is not limited to, fighting, being in trouble, and giving a speech. For space, a particle system can be used to render meteor trail particles and nebula particles, and an animation system can be used to render meteor trail animations and nebula animations. For the ocean, a particle system can be used to render water flow particles and bubble particles, and an animation system can be used to render water flow animations and bubble animations. For fireworks, a particle system can be used to render fireworks particles, and an animation system can be used to render fireworks animations. For fighting, a particle system can be used to render light and shadow particles, and an animation system can be used to render light and shadow animations to achieve light and shadow effects. An explosion particle system can also be used to render explosion animations to achieve explosion effects. For a speech, an animation system can be used to render text animations to achieve text effects.

[0081] To better understand the embodiments of this application, the structure of the electronic device of this application is described below:

[0082] Figure 3 A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0083] Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0084] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0085] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0086] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0087] In this embodiment, the camera 193 can acquire image data and display the image on the display screen 194. The image displayed on the display screen 194 can have particle rendering and / or animation rendering added. The processor 110 can analyze the scene and characters in the image and add particle rendering and / or animation rendering according to the scene and characters in the image.

[0088] Figure 4 A schematic diagram of the software structure of the electronic device 100 is shown.

[0089] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0090] Figure 4This is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into multiple layers, from top to bottom: the application layer, the application framework (FWK) layer, the Android runtime and system libraries, and the kernel layer.

[0091] The application layer can include a series of application packages.

[0092] like Figure 4 As shown, the application package may include video parsing modules, graphics engine rendering modules, AI inference analysis modules, effects enhancement modules, and applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0093] The video parsing module is responsible for decoding the video file and outputting video frames. During decoding and output, relevant frame change events are notified. These events are provided to the video frame analysis module, allowing it to detect the generation of new video frames and perform inference and analysis. A video frame is also called a video image, or simply an image.

[0094] The AI ​​inference and analysis module is responsible for analyzing video frames to obtain their feature information. For example, it can derive feature information for the current video frame based on a dataset trained on a large AI model. This feature information can include scene information, character information, and depth information. Scene information, also known as scene characteristics, can include location and weather. Location can be in space, ocean, etc., and weather can include rain, snow, thunder, etc. Character information, also known as task characteristics, can include whether a character is fighting or in a predicament. Depth information indicates the depth of various objects in the video frame. The analysis results of scene characteristics, character characteristics, and depth information from the AI ​​inference and analysis module are then provided to the effects enhancement module.

[0095] The effects enhancement module receives feature information from the AI ​​inference and analysis module, selectively activates particle systems and animation systems based on scene and character characteristics, and adjusts the characteristic parameters and control flow logic of these systems. For example, particle system characteristics can include particle speed and quantity. The control flow logic can include particle rendering start time and duration. The effects enhancement module's rendering can be applied to the graphics engine rendering module, outputting the image displayed on the electronic device. The particle system in the effects enhancement module performs particle rendering based on scene characteristics, character characteristics, and depth information provided by the AI ​​inference and analysis module. Particle rendering can include 2D / 3D particle rendering. The animation system in the effects enhancement module performs animation rendering based on scene characteristics, character characteristics, and depth information provided by the AI ​​inference and analysis module.

[0096] The graphics engine rendering module is responsible for blending video frame rendering, particle rendering, and animation rendering. After acquiring video frames from the video parsing module, the graphics engine rendering module renders the video frames on the background layer, using the rendered video frames as the background. It then uses particle rendering and animation rendering as the foreground layer, blending the foreground and background layers to obtain the desired display image, which can be called the target image or the final image. The graphics engine rendering module can also be called the rendering blending module.

[0097] It should be understood that the particle system can perform particle rendering, the animation system can perform animation rendering, and then the image engine rendering module can add particle rendering and animation rendering based on the depth information. Alternatively, a depth module can be added between the effects enhancement module and the image engine rendering module. This depth module can then add particle rendering and animation rendering based on the depth information, and the image engine rendering module can then fuse the results of the particle rendering and animation rendering added by the depth module with the video frames.

[0098] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0099] like Figure 4 As shown, the application framework layer may include a display FWK, a camera FWK, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0100] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0101] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0102] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0103] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0104] The Hardware Abstraction Layer (HAL) is an interface layer located between the operating system kernel and upper-level software. Its purpose is to abstract hardware. The HAL is an abstract interface for device kernel drivers, used to provide application programming interfaces for accessing the underlying device to higher-level Java API frameworks. The HAL contains several library modules, such as the Camera HAL and the Display HAL.

[0105] Each library module implements an interface for a specific type of hardware component. For example, the Camera HAL provides the Camera FWK with an interface to access hardware components such as the camera lens. The Display HAL provides the Display FWK with an interface to access hardware components such as the display screen. When the system framework layer API requires access to the portable device's hardware, the Android operating system loads the library module for that hardware component.

[0106] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0107] The following example illustrates the workflow of electronic device software.

[0108] For example, the camera driver can sequentially transmit image data (also known as video frame data) captured by the camera through the camera HAL and camera FWK to the camera application. The camera application then transmits this image data to the video parsing module, which decodes the image data to obtain the image to be displayed, also known as a video frame. The video parsing module then transmits the image to be displayed to both the AI ​​inference analysis module and the graphics engine rendering module. The AI ​​inference analysis module analyzes the image to be displayed, identifying scene features, character features, and depth information, and transmits these features to the effects enhancement module. The effects enhancement module's particle system then performs particle rendering based on these features, and its animation system performs animation rendering. The particle rendering result is transmitted to the graphics engine rendering module via the particle system, and the animation rendering result is transmitted to the graphics engine rendering module via the animation system. Then, the image engine rendering module renders the image to be displayed, and blends the rendering result of the image to be displayed as the background layer, the particle rendering result, and the animation rendering result as the foreground layer to obtain the target image. The target image can be displayed on the monitor in sequence through the camera application, camera FWK, display FWK, display HAL, and display driver control. The displayed target image will then have particle rendering effects and animation rendering effects added.

[0109] It should be understood that the image data transmission method and the target image transmission method in the above examples can be configured as needed and are not limited to the transmission methods in the examples above. For example, a first transmission channel can be added between the camera driver and the video parsing module, through which the camera driver can directly transmit image data to the video parsing module. Alternatively, a second transmission channel can be added between the graphics engine rendering module and the display FWK, through which the graphics engine rendering module can directly transmit the target image to the display FWK. In addition, the division of each module and the processing of each module can also be configured as needed. For example, the processing of the video parsing module, the AI ​​inference analysis module, the effect enhancement module, and the graphics engine rendering module can be performed through one module, without any restrictions.

[0110] The following section, with reference to some exemplary user interface diagrams, details the shooting scenarios provided in this application.

[0111] It is understood that the terms "interface" and "user interface" in the embodiments of this application refer to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and a form acceptable to the user. A commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, button, or other interface element displayed on the screen of an electronic device. Buttons can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visible interface elements.

[0112] For ease of understanding, this application provides an example of taking photos / recording videos on a rainy day.

[0113] like Figure 5 As shown, Figure 5 This is a schematic diagram of a shooting scene provided in an embodiment of this application.

[0114] like Figure 5 As shown in (a), the user interface (also known as the main interface) displays a page with application icons. This page may include multiple application icons (e.g., camera app, weather app, calendar app, photo album app, notes app, email app, app store app, settings app, etc.). Below these application icons, a page indicator may also be displayed to show the positional relationship between the currently displayed page and other pages. Below the page indicator are multiple application icons (e.g., camera app icon 101, browser app icon, messaging app icon, dialer app icon). These application icons remain displayed when switching pages.

[0115] As can be understood, camera app icon 101 is the icon for the camera application (i.e., the camera app). Camera app icon 101 can be used to trigger the launch of the camera application.

[0116] In this embodiment, the electronic device can detect user actions applied to the camera application icon 101. In response to these actions, the electronic device can display an initial shooting preview interface, such as... Figure 5 The user interface shown in (b) is shown in the image.

[0117] It is understood that the user operations mentioned in this application may include, but are not limited to, touch (e.g., click), voice control, gestures, etc., and this application does not limit them.

[0118] like Figure 5As shown in (b), the user interface is the default shooting interface of the camera application, where the user can preview images. The user interface may include a preview window 201, camera mode options 202, album shortcut buttons, and a shutter button 203 (also known as a shooting button). For example, the preview image displayed on the user interface may include a first person and a second person. For example, the first person may be a female person, and the second person may be a male person.

[0119] The preview window 201 of the user interface can be used to display a preview image. The preview image displayed in the preview window 201 of the user interface can be an image captured by the camera of the electronic device based on the field of view when the photo mode option is selected.

[0120] Camera mode option 202 may display one or more shooting mode options. These shooting mode options may include: aperture mode option, night scene mode option, intelligent portrait mode option, photo mode option, rendering mode option 2021, movie mode option, and more options. In another possible implementation, the mode option in camera mode 202 may be switched when a sliding operation is detected on camera mode option 202.

[0121] Understandably, camera mode option 202 may include more or fewer shooting mode options.

[0122] In this embodiment, the electronic device can detect a user operation applied to the rendering mode option 2021. In response to this operation, the electronic device can display a video recording interface, such as... Figure 5 The user interface shown in (c) is shown in the image.

[0123] like Figure 5 As shown in (c), the user interface is the interface for entering the rendering mode, and the rendering mode option 2021 is selected. In the user interface, entering the rendering mode allows the addition of particle rendering 207 related to the image displayed in the preview window 201, such as raindrop particle rendering 207. Furthermore, the user interface can also add animation rendering 208 related to the image displayed in the preview window 201; this animation rendering 208 could be, for example, cloud animation rendering 208.

[0124] It should be understood that the particle rendering 207 and animation rendering 208 in the embodiments of this application can be particle rendering 207 and animation rendering 208 for the same target, that is, particle rendering 207 and animation rendering 208 are performed separately for the same target, such as performing particle rendering 207 and animation rendering 208 for raindrops. In addition, the particle rendering 207 and animation rendering 208 in the embodiments of this application can also be performed separately for different targets, that is, particle rendering 207 is performed for one target and animation rendering 208 is performed for another target, such as performing particle rendering 207 for raindrops and animation rendering 208 for clouds, and no limitation is made here.

[0125] Then, as Figure 5 As shown in (d), the electronic device can detect user operation on the shutter button 203 and start recording video. During the recording process, as... Figure 5 As shown in (e), the first and second characters can move during recording. For example, if the second character moves out of the electronic device's field of view, then the following can be displayed: Figure 5 As shown in (e), Figure 5 The image in (e) includes the first person but excludes the second person.

[0126] Then, as Figure 5 As shown in (f), if the electronic device detects a user operation that is applied to the shutter button 203 again, a video can be generated. The video can be saved in the photo album. When the electronic device detects a user operation that is applied to the photo album shortcut button, it can display the video. The start time of the video can be as follows: Figure 5 As shown in (d) above, the time at which the shutter button 203 is activated can be the end time of the video, which can be as follows: Figure 5 The time at which the shutter button 203 is detected again, as shown in (f) in the video, can optionally include the following: Figure 5 (d) Figure 5 (e) and Figure 5 The image of the interface shown in (f) is shown in the image.

[0127] In this embodiment of the application, after entering the rendering mode, particle rendering 207 and animation rendering 208 are added to the displayed image and the recorded video, which can improve the visual effect of the image displayed by the electronic device, and the recorded video can also be rendered with rendering effects, which can also improve the rendering effect of the video.

[0128] It should be noted that for the camera function of an electronic device, image rendering effects can also be added. In one possible implementation, in rendering mode, a click operation on the shutter button 203 can trigger the electronic device to take a picture, and the captured image can also have particle effects and animation effects added. Furthermore, a long press operation on the shutter button 203 can trigger the electronic device to play video. In this case, the start time of the video can be the start time of the long press operation, that is, the time when the electronic device begins to detect the action on the shutter button 203, and the end time of the video can be the end time of the long press operation, that is, the time when the electronic device begins to detect the action on the shutter button 203.

[0129] In the above embodiment, rendering effects are enabled by default upon entering rendering mode, such as enabling particle rendering 207 and animation rendering 208. Optionally, rendering effects can remain enabled in rendering mode. In another possible implementation, the enabling or disabling of particle rendering 207 and animation rendering 208 can also be controlled.

[0130] The following example illustrates how to control the on / off state of particle rendering 207 and animation rendering 208 in a camera mode.

[0131] After launching the camera application, it can display something like this: Figure 6 The interface shown in (a) is shown in the image. Figure 6 In the interface shown in (a), camera mode option 202 includes portrait mode 2022. The image displayed in preview window 201 has no rendering effects added. In portrait mode 2022, which can be a shooting mode designed for photographing people, this mode optimizes camera settings and algorithms to provide more vivid and natural portrait photos. Figure 6 In (a) of the image, the preview window 201 displays an image without any rendering effects. The electronic device can detect user actions applied to Portrait Mode 2022 and can then enter... Figure 6 The interface shown in (b) is shown in the image.

[0132] Figure 6 The interface shown in (b) is the display interface corresponding to Portrait Mode 2022, and Portrait Mode 2022 is in the selected state. Figure 6In the interface shown in (b), the image displayed in the preview window 201 also lacks rendering effects. This means that rendering effects are not enabled by default when the electronic device enters portrait mode. The display interface corresponding to portrait mode also includes a rendering control button 204. This rendering control button 204 can be used to control the simultaneous on / off of rendering effects, or it can be described as a one-click on / off function for rendering effects. For example, this rendering control button 204 can be used to control the on / off of all rendering effects. Assuming the rendering effects include particle rendering 207 and animation rendering 208, then this rendering control button 204 can be used to control the simultaneous on / off of particle rendering 207 and animation rendering 208, and also to control the simultaneous off / off of particle rendering 207 and animation rendering 208.

[0133] It should be noted that the location of the rendering control button 204 can be set as needed and is not limited here. Optionally, the location of the rendering control button 204 can be fixed or adjustable. For example, adjusting the rendering control button 204 can be done by dragging the rendering control button 204. This embodiment allows users to adjust the position of the rendering control button 204 according to their needs or usage habits, thereby improving the user experience.

[0134] like Figure 6 As shown in (b), when an electronic device enters portrait mode, rendering effects are not enabled by default. In another possible implementation, when an electronic device enters portrait mode 2022, rendering effects may be enabled by default, such as enabling at least one of particle rendering 207 and animation rendering 208 by default, which is not limited here.

[0135] Then, as Figure 6 As shown in (c), the electronic device can detect user operation on the rendering control button 204 and can display, as shown in [the diagram]. Figure 6 The interface shown in (d) is shown in the figure. Figure 6 (c) and Figure 6 (b) in the text can be a sequential relationship, such as displaying first. Figure 6 (b) is displayed again. Figure 6 (c) in. Figure 6 In the interface shown in (d), the image displayed in the preview window 201 has added rendering effects, such as particle rendering 207 and animation rendering 208. That is to say, the electronic device can add rendering effects after detecting a user operation on the rendering control button 204 in an interface without rendering effects.

[0136] Then, as Figure 6As shown in (e), the electronic device can detect user operation on the rendering control button 204, and can then display as shown in Figure (e). Figure 6 The interface shown in (f) is as follows. Among them, Figure 6 (e) in Figure 6 (d) in the text can be a sequential relationship, such as displaying first. Figure 6 (d) is displayed again. Figure 6 (e) in the example. Figure 6 In the interface shown in (f), the image displayed in the preview window 201 has no rendering effect. That is to say, the electronic device can remove the rendering effect after detecting a user operation on the rendering control button 204 in the interface where rendering effects are added.

[0137] In another possible implementation, the rendering effect can be turned on or off via different control buttons in the interface corresponding to Portrait Mode 2022. For example, one control button can be used to turn the rendering effect on, and another control button can be used to turn it off. In this embodiment, controlling the rendering effect via the rendering control button 204 can reduce the number of elements displayed on the interface and improve the simplicity of the display interface.

[0138] It should be understood that the rendering control button 204 can also be added in other modes, such as adding the rendering control button 204 in video mode or photo mode, and there are no restrictions here.

[0139] It should be noted that rendering effects can be turned on or off before video recording, or during video recording. Optionally, if rendering effects are turned off during recording, the recorded video will include images with rendering effects disabled. For example, if the video recording time is from T1 to T3, and rendering effects are turned off at time T2, then the video from T1 to T2 will have rendering effects added, while the video from T2 to T3 will not have rendering effects added. As another example, assuming rendering effects are turned on again at time T4, then the video from T1 to T2 will have rendering effects added, while the video from T2 to T4 will not have rendering effects added, and the video from T4 to T3 will have rendering effects added.

[0140] It should be understood that T1 < T2 < T4 < T3.

[0141] This application embodiment improves the user experience by allowing users to select the time period for enabling or disabling the rendering effect by controlling its on / off state. Furthermore, the rendering control button 204 provides convenient one-click on / off control of the rendering effect.

[0142] Figure 6 The scene shown is Figure 5 Compared to the scenarios shown, Figure 6 Added rendering control buttons 204 to enable / disable rendering effects.

[0143] The above embodiment illustrates the one-click on and one-click off of rendering effects. In another possible implementation, different rendering effects can be controlled by different buttons. For example, particle rendering 207 can be controlled by one button, and animation rendering 208 can be controlled by another button.

[0144] like Figure 7 The interface shown in (a) is the interface after entering portrait mode. Figure 7 In the interface shown in (a), the image displayed in preview window 201 has no rendering effect, and Figure 7 The interface shown in (a) includes a particle control button 205 and an animation control button 206. The particle control button 205 is used to control the particle rendering 207 to be turned on or off, and the animation control button 206 is used to control the animation rendering 208 to be turned on or off.

[0145] Then, as Figure 7 As shown in (b), the electronic device can detect user operation on the animation control button 206 and display as shown in [the diagram]. Figure 7 The interface shown in (c) is as follows. Figure 7 (b) and Figure 7 (a) in the text can be a sequential relationship, such as displaying first. Figure 7 (a) is displayed again. Figure 7 (b) in the middle. Figure 7 In the interface shown in (c), the image displayed in the preview window 201 has animation rendering 208 added but particle rendering 207 not added. That is to say, the electronic device can add animation rendering 208 after detecting a user operation on the animation control button 206 in the interface without animation rendering 208.

[0146] Then, as Figure 7 As shown in (d), the electronic device can detect user operation on the particle control button 205 and display as shown in Figure 205. Figure 7 The interface shown in (e) is as follows. Figure 7 (d) in Figure 7 (c) in the text can be a sequential relationship, such as displaying first. Figure 7 (c) is displayed again. Figure 7 (d) in. Figure 7 In the interface shown in (e), the image displayed in the preview window 201 has particle rendering 207 and animation rendering 208 added. That is to say, the electronic device can add particle rendering 207 after detecting a user operation on the particle control button 205 in an interface where particle rendering 207 has not been added.

[0147] Then, as Figure 7 As shown in (f), the electronic device can detect user operation on the animation control button 206 and display as shown in (f). Figure 7 The interface shown in (g) is as follows. Among them, Figure 7 (f) in Figure 7 (e) in the text can be a sequential relationship, such as displaying first. Figure 7 (e) is displayed again. Figure 7 (f) in. Figure 7 In the interface shown in (g), the image displayed in the preview window 201 has no animation rendering 208 but has particle rendering 207. That is to say, after the electronic device detects a user operation on the animation control button 206 in the interface with added animation rendering 208, it can remove the animation rendering 208.

[0148] Then, as Figure 7 As shown in (h), the electronic device can detect user operation on the particle control button 205 and display as shown in [the diagram]. Figure 7 The interface shown in (i) is as follows. Figure 7 (h) in Figure 7 (g) in the text can represent a sequential relationship, such as displaying first. Figure 7 (g) is displayed again. Figure 7 (h) in. Figure 7 In the interface shown in (i), the image displayed in the preview window 201 has neither animation rendering 208 nor particle rendering 207 added. That is to say, after the electronic device detects a user operation on the particle control button 205 in the interface with particle rendering 207 added, it can remove particle rendering 207.

[0149] Optionally, the buttons in this embodiment can be in a fixed position or an adjustable position. For example, the particle control button 205 and the animation control button 206 can be in a fixed position or an adjustable position. Please refer to the description of the rendering control button, which will not be repeated here.

[0150] It should be noted that rendering effects can be turned on or off before video recording, or during video recording. Optionally, if rendering effects are turned off during recording, the recorded video will include images with rendering effects disabled. For example, if the video recording time is from T5 to T10, then particle rendering 207 is turned off at time T6, particle rendering 207 is turned on again at time T7, animation rendering 208 is turned off at time T8, and animation rendering 208 is turned on again at time T9. In the resulting video, the video from T5 to T6 has both particle rendering 207 and animation rendering 208 added, while the video from T6 to T7 does not have particle rendering 207 added but has animation rendering 208 added. Then, the video from T7 to T8 has both particle rendering 207 and animation rendering 208 added, the video from T8 to T9 has particle rendering 207 added but not animation rendering 208 added, and the video from T9 to T10 has both particle rendering 207 and animation rendering 208 added.

[0151] It should be understood that T5 < T6 < T7 < T8 < T9 < T10.

[0152] It should be noted that after entering rendering mode, you can either turn off particle rendering 207 and animation rendering 208 by default, or turn on particle rendering 207 and turn off animation rendering 208 by default, or turn off particle rendering 207 and turn on animation rendering 208 by default. You can set the default rendering effects added or not added after entering rendering mode as needed, and there are no restrictions here.

[0153] In one possible implementation, the default rendering effects added or not added upon entering rendering mode can be set based on the usage parameters of each rendering effect. These parameters can include usage duration or usage count. For example, the default display could show rendering effects with a usage duration exceeding a duration threshold, and / or rendering effects with a usage count exceeding a count threshold. This allows for setting the default rendering effects added or not added upon entering rendering mode based on user habits, thus improving the user experience.

[0154] In this embodiment, different buttons can be used to control the on or off of different rendering effects, so that the rendering effects to be turned on or off can be selected as needed, which can improve the flexibility of adding rendering effects to the acquired image.

[0155] It should be understood that the operation of controlling the rendering effect to be turned on or off in the above embodiments can also be achieved through other operations, and is not limited to user operations on buttons. It is understood that the shooting scene in the above examples can be a shot of a landscape or an animal, and is not limited to a shot of a person; this will not be elaborated upon here. Furthermore, the actual rendering effect can be only one type, such as only particle rendering 207 or animation rendering 208. Moreover, the rendering effect is not limited to particle rendering 207 and animation rendering 208; various rendering effects can be added as needed, and no specific rendering effect is limited here.

[0156] Figure 7 The scene shown is Figure 6 Compared to the scenarios shown, Figure 7 Each of the corresponding control buttons is set to control the on or off of different rendering effects, such as particle control button 205 and animation control button 206 to control particle rendering on or off.

[0157] It should be understood that rendering control buttons 204, particle control buttons 205, and animation control buttons 206 can also be set in the interface. This way, users can choose to turn all rendering effects on or off, or turn on or off some rendering effects as needed. This can improve the flexibility of users in turning rendering effects on or off and improve the user experience.

[0158] It should be understood that rendering could also be omitted from the interface during shooting, with the rendering effect added when the captured image is saved, allowing the electronic device's gallery to display the image with the added rendering effect. However, this embodiment adds the rendering effect to the image displayed during shooting, enabling users to predict the final image's effect while shooting, thereby improving the user experience.

[0159] It should be understood that the above user interface is merely an example provided in this application and should not be considered as a limitation of this application.

[0160] The specific implementation of the embodiments of this application will be illustrated below.

[0161] Please see Figure 8 , Figure 8 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 8 The methods shown may include:

[0162] S801, the video parsing module receives video frame data.

[0163] Among them, video frame data can be data collected by the camera of an electronic device during the video recording process.

[0164] In another possible implementation, the video frame data can also be image data, which can be data collected by the camera of an electronic device during the process of taking a picture.

[0165] S802, the video parsing module decodes the video frame data to obtain the video frame.

[0166] In this context, a video frame can also be called an image frame, an image, or a picture. For example, a video frame can be, for instance, a... Figure 5 The image displayed in preview window 201 shown in (b) of the image.

[0167] The S803 and video parsing modules transmit video frames to the graphics engine rendering module and the AI ​​inference analysis module, respectively.

[0168] It should be noted that, in this embodiment of the application, the video parsing module can transmit video frames to the graphics engine rendering module and the AI ​​inference analysis module simultaneously or in shifts.

[0169] The S804 AI inference and analysis module analyzes video frames to obtain target scene information, target object information, and target depth information of the video frames.

[0170] Scene information can represent the scene of an image. In this embodiment, target scene information can represent the scene of a video frame. The scene can include, but is not limited to, location and weather. Location can include, but is not limited to, space and ocean. Weather can include, but is not limited to, afternoon and thunder. Optionally, target scene information can include a target scene identifier. The target scene identifier can include a target location identifier and a target weather identifier. Object information can represent the action of an object in the image. The object can be a living being such as a person or animal, or a non-living being such as a robot, without limitation. In this embodiment, target object information can represent the action of a target object in a video frame. Optionally, target object information can include a target action identifier. The target object can be one of all objects in the video frame. Actions can include, but are not limited to, fighting and speaking. Depth information can be the distance from any object point in the three-dimensional space of the real scene to the camera. Depth information can be represented in the image by a depth image. The pixel value of each pixel in the depth image reflects the depth of the corresponding point in the real scene. In this embodiment, target depth information can represent the depth of each pixel in the video frame.

[0171] In this embodiment, optionally, the AI ​​inference analysis module can input video frames into a trained model, and the trained model can output target scene information and target object information of the video frames. Optionally, the trained model can be, for example, a large language model. In one possible implementation, the trained model can be trained using supervised learning. For example, image samples are used as input to the model to be trained. Then, the inference scene information obtained by the model processing the image samples is compared with the scene labels corresponding to the image samples. If the comparison is inconsistent, the model parameters are updated and training continues. The inference object information obtained by the model processing the image samples is compared with the object action labels corresponding to the image samples. If the comparison is inconsistent, the model parameters are updated and training continues until the inference scene information obtained by the model processing the image samples matches the scene labels corresponding to the image samples and the inference object information obtained by the model processing the image samples matches the object action labels corresponding to the image samples. At this point, the training of the model is considered complete, and a trained model is obtained. Optionally, the target scene information is one of multiple scene labels. The target object information is one of multiple object action tags.

[0172] Optionally, the target depth information can be obtained by the camera when capturing video frames, such as obtaining the depth of video frames when capturing video frames using a binocular camera. Alternatively, the target depth information of video frames can be obtained through model inference.

[0173] S805, the AI ​​inference and analysis module transmits target scene information, target object information, and target depth information to the effect enhancement module.

[0174] S806 The effect enhancement module uses the first mapping relationship to obtain the rendering effect corresponding to the target scene information, and uses the second mapping relationship to obtain the rendering effect corresponding to the target object information.

[0175] The first and second mapping relationships can be pre-configured. The first mapping relationship represents the relationship between scene information and rendering effects. For example, the first mapping relationship may include, but is not limited to, space corresponding to meteor trails and nebula effects, ocean corresponding to water flow and bubbles, rain corresponding to raindrops, and thunder corresponding to lightning. Space corresponding to meteor trails and nebula effects means that if the identified scene is space, particle rendering and animation of meteor trails and nebula effects can be performed. Ocean corresponding to water flow and bubbles means that if the identified scene is ocean, particle rendering and animation of water flow and bubbles can be performed. Rain corresponding to raindrops means that if the identified scene is raining, particle rendering and animation of raindrops can be performed. Thunder corresponding to lightning means that if the identified scene is thundering, particle rendering and animation of lightning can be performed. No further restrictions are imposed.

[0176] The second mapping relationship is used to represent the relationship between object information and rendering effects. For example, the second mapping relationship may include, but is not limited to, fighting corresponding to lighting and shadow effects, and speeches corresponding to text effects. If fighting corresponds to lighting and shadow effects, then the identified target object information is fighting, and particle rendering and animation rendering of lighting and shadow effects can be performed; if speeches correspond to text effects, then the identified target object information is speeches, and animation rendering of text effects can be performed.

[0177] It should be noted that the rendering in this embodiment can be for existing targets in the video frame, such as raindrops in the video frame. Alternatively, it can be for newly added targets in the video frame. For example, if the video frame contains raindrops but not lightning, then a lightning rendering effect can be added to the video frame.

[0178] It should be understood that the embodiments of this application can improve the realism of the rendered video frames by rendering existing targets in the video frames. Furthermore, the embodiments of this application can enrich the elements in the video frames by rendering newly added targets.

[0179] S807, the effect enhancement module generates the rendering effect corresponding to the target scene information, and generates the rendering effect corresponding to the target object information.

[0180] The rendering effect of this embodiment can be referred to the description in S806, and will not be repeated here. Optionally, the effect enhancement module can call the particle system to generate particle rendering corresponding to the target scene information and particle rendering corresponding to the target object information, and call the animation system to generate animation rendering corresponding to the target scene information and animation rendering corresponding to the target object information.

[0181] It should be noted that if the rendering in this embodiment is for an existing target in a video frame, then the position of the rendering effect in this embodiment is the position of the rendered target. If the rendering is for a newly added target in a video frame, the area where the rendering effect is located can be random or determined according to a set rule. Regardless of how the area where the rendering effect is determined, the area where the rendering effect is located can be recorded.

[0182] S808, the effect enhancement module uses the target depth information to generate L-level depth information.

[0183] In this embodiment, the Kth level depth information in the L-level depth information is related to at least two K-1 level depth information in the L-level depth information, 1 < K ≤ L, and K and L are integers. Please refer to [link to relevant documentation]. Figure 9 , Figure 9 This is a schematic diagram of multi-level depth information provided in an embodiment of this application. Since the next level of depth information of L-level depth information is obtained by using the previous level of depth information, it can also be understood that the next level of depth information of L-level depth information is obtained by compressing the previous level of depth information. Therefore, the depth information in the embodiment of this application can also be called depth compressed information.

[0184] like Figure 9 As shown, the previous level of depth information can be compressed to obtain the next level of depth information. For example, the first level of depth information may include the depth of each pixel in a video frame. Figure 9 The first-level depth information shown may include the depths of each of the 8*8=64 pixels in the video frame. The second-level depth information is obtained based on the first-level depth information; for example, the depth of any second region in the second-level depth information is the minimum depth among at least two pixels in the first-level depth information. The third-level depth information is obtained based on the second-level depth information; for example, the depth of any third region in the third-level depth information is the minimum depth among at least two second regions in the second-level depth information. The fourth-level depth information is obtained based on the third-level depth information; for example, the depth of any fourth region in the fourth-level depth information is the minimum depth among at least two third regions in the third-level depth information. Optionally, at least two pixels are adjacent pixels, and at least two regions can be adjacent regions. It should be understood that a pixel can be understood as a region.

[0185] For example, the minimum depth value among every four pixels (also referred to as the first region) in the first-level depth information is used as the depth value of one of the second regions in the second-level depth information. Thus, one or more depth values ​​of the second regions can be obtained, meaning the second-level depth information includes the depth values ​​of one or more second regions. Then, if there are multiple second regions, the minimum depth value among every four second regions in the second-level depth information is used as the depth value of one of the third regions in the third-level depth information. Thus, one or more depth values ​​of the third regions can be obtained, meaning the third-level depth information includes the depth values ​​of one or more third regions. Then, if there are multiple third regions, the minimum depth value among every four third regions in the third-level depth information is used as the depth value of one of the fourth regions in the fourth-level depth information. Thus, one or more depth values ​​of the fourth regions can be obtained. Optionally, every four regions can be considered a group of regions, and different groups of regions do not include the same regions. For example, the first group of regions includes pixels 1, 2, 3, and 4, and the second group of regions includes pixels 5, 6, 7, and 8. In another possible implementation, the minimum depth value can also be replaced with the average or median, etc., which can be set as needed and is not limited here.

[0186] It should be understood that in the embodiments of this application, the number of parent regions used to determine the depth value of the next level region can be set as needed, and is not limited to the four regions in the example above. Furthermore, the number of parent regions used to determine the depth information at different levels can also be different. For example, in another possible implementation, every two values ​​in the first level depth information can also be referred to as the minimum depth value in the first region, serving as the depth value of one of the second regions in the second level depth information, and every three minimum depth values ​​of the second regions in the second level depth information serve as the depth value of one of the third regions in the third level depth information.

[0187] S809, The effect enhancement module obtains the depth value corresponding to the region where the rendering effect is located from the L-level depth information.

[0188] In this embodiment, the L-th level depth information is the last level of depth information within the L-th level depth information. This last level of depth information contains the fewest depth values ​​among all the L-th level depth information. The first level of depth information contains the most depth values, meaning the number of depth values ​​decreases sequentially from the first level to the L-th level. In this embodiment, the depth information includes the depth values ​​of each region, allowing the depth values ​​of the region where the rendering effect is located to be obtained from the depth information.

[0189] S810, the effect enhancement module determines whether the depth of the rendering effect is less than the depth value corresponding to the area where the rendering effect is located.

[0190] The depth of the rendering effect can be the depth at which the rendering effect is added to the video frame.

[0191] In this step, if the depth of the rendering effect is less than the depth value of the area where the rendering effect is located, then execute S813; if not, that is, if the depth of the rendering effect is not less than the depth value of the area where the rendering effect is located, then execute S811.

[0192] S811, the effect enhancement module determines whether L is greater than 1.

[0193] In this step, if L is greater than 1, there are still some levels of depth information that have not been compared with the depth of the rendering effect, so S814 is executed.

[0194] S812, Let L = L-1.

[0195] In this embodiment, setting L = L-1 aims to compare the depth of the rendered effect with the depth of the region where the rendered effect is located in the (L-1)th level depth information. After setting L = L-1, the process can return to step S809. This allows the (L-1)th level depth information to be obtained, and then the depth value corresponding to the region where the rendered effect is located can be obtained from the L-level depth information. Finally, it is determined whether the depth of the rendered effect is less than the depth value corresponding to the region where the rendered effect is located.

[0196] It should be understood that S809-S813 is a process of comparing the depth information of each level in the L-level depth information with the depth of the rendering effect. This comparison can obtain the rendering area of ​​the video frame. The comparison can start from the last level of depth information, comparing the depth of each level of depth information with the depth of the rendering effect. If the depth of one region in the K-th level depth information is greater than the depth of the rendering effect, the rendering effect performed in that region will not be occluded by the target in that region, and therefore rendering can be performed in that region. If the depth of one K-th region in the K-th level depth information is less than the depth of the rendering effect, the rendering effect performed in that K-th region will be occluded by the target in that K-th region. Therefore, the depths of the K-th region in the K-th level depth information and the corresponding K-1 regions in the (K-1)-th level depth information can be compared. It should be noted that if the depth of one first region in the first level depth information is less than the depth of the rendering effect, no rendering effect is added to that first region, thereby reducing the size of the image data.

[0197] In summary, S809-S813 traverses the depth compression information from high to low level, obtains the minimum depth value corresponding to the current region where the current rendering effect (such as the current particle, the current animation, etc.) is located, and then compares the minimum depth value corresponding to the current region with the depth of the rendering effect to determine whether the target in the current region will occlude the rendering effect. If it does not occlude, the current region is a safe active region; if it does occlude, the previous level of depth compression information is traversed.

[0198] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating the comparison of multi-level deep compression information provided in an embodiment of this application. Figure 10 The rendering effect includes particle rendering as an example. It should be understood that animation rendering can also adopt the solution of the embodiments of this application.

[0199] like Figure 10 The methods shown may include:

[0200] S1001, Traverse the depth-compressed information from high to low.

[0201] In this embodiment of the application, traversing the depth compression information from high to low can start from the last level of depth compression information, for example, starting from the Lth level of depth compression information.

[0202] S1002, Obtain the minimum depth of the region where the current particle is located.

[0203] The current particle can be the particle that needs to be depth compared. In this embodiment, if it starts from the last level of depth compression information, the minimum depth value corresponding to the region where the current particle is located is first obtained from the last level of depth information.

[0204] S1003. Compare the minimum depth with the current particle's depth.

[0205] In this embodiment, the minimum depth is compared with the current particle's depth to determine whether the current particle will be occluded in its region, for example, whether the current particle will be occluded by a target in its region. If occlusion exists, the process returns to step S1001; if no occlusion exists, the process proceeds to step S1004.

[0206] S1004: The area where the current particle is located is updated to a safe activity area. Particles within this area no longer need to be compared in depth.

[0207] In this embodiment, if there is no occlusion, the area where the current particle is located is updated to a safe activity area, which means that the current particle can be rendered in the safe activity area.

[0208] For example, in combination Figure 9 The depth comparison process of this application embodiment is illustrated with an example. For example... Figure 9 As shown, the fourth region where the rendering effect is located can be obtained. Then, the depth of the rendering effect is compared with the depth value of the fourth region where the rendering effect is located in the fourth-level depth information. If the depth value of the rendering effect is greater than or equal to the depth value of the fourth region where the rendering effect is located, the depth of the rendering effect is compared with the third-level depth information. At this time, the depth value of the third region where the rendering effect is located is obtained, and then compared with the depth value of the third region where the rendering effect is located. If the depth value of the rendering effect is less than the depth value of the third region where the rendering effect is located, the third region is designated as a safe active region, that is, the rendering effect is added to the safe active region. If the depth value of the rendering effect is greater than or equal to the depth value of the third region where the rendering effect is located, the depth of the rendering effect is compared with the second-level depth information, and so on, until the comparison between the first-level depth information and the depth value of the rendering effect is completed.

[0209] S813, the effect enhancement module updates the area where the rendering effect is located to the rendering area.

[0210] The rendering region can be the area where rendering effects are added. The rendering region can also be called the safe active region, where adding rendering effects will not be obscured by any target within that safe active region of the image.

[0211] S814, the effects enhancement module adds particle rendering and animation rendering to the rendering area.

[0212] For example, the particle rendering and animation rendering effects of the embodiments of this application can be referred to Figure 5 Examples are provided, and limitations are not specified. Optionally, no rendering effects may be added to non-safe activity areas, which can reduce the size of the image data.

[0213] S815, the effects enhancement module transmits the particle rendering results and animation rendering results to the graphics engine rendering module.

[0214] The S816 graphics engine rendering module renders video frames and merges the rendered video frames, particle rendering results, and animation rendering results to obtain the target image.

[0215] In this embodiment, rendering video frames can involve, for example, super-resolution processing, lighting processing, and shading processing. Super-resolution processing can increase the resolution of video frames, while lighting processing can simulate the illumination effects of natural or artificial light on virtual scenes or objects, including direct light, diffused light, and reflected light, to enhance the three-dimensionality and realism of the scene. Shading processing can assign color to each pixel or object surface in the scene based on the material, texture, and lighting conditions of the object, making the image more vivid and realistic.

[0216] For example, the target image in an embodiment of this application can be, for example, as follows: Figure 5 As shown in (c) in the figure.

[0217] In another possible implementation, if the rendering effect can be turned on or off via a rendering control button, then steps S803-S816 in this embodiment can be executed after the rendering effect is turned on via the rendering control button. In this case, steps S803-S816 are used to process the video frames captured by the electronic device. This application executes steps S803-S816 after the rendering effect is turned on via the rendering control button, which saves image processing computing resources and can generate the video / image frames required by the user.

[0218] In another possible implementation, particle rendering or animation rendering can be performed by default, which can reduce the computing resources required by electronic devices during rendering.

[0219] In another possible implementation, if particle rendering and animation rendering are controlled to be turned on or off via different buttons—for example, a particle control button to turn particle rendering on or off, and an animation control button to turn animation rendering on or off—then particle rendering is added to the rendering area only when particle rendering is turned on via the particle rendering control button, and not when particle rendering is turned off via the particle rendering control button. Similarly, animation rendering is added to the rendering area only when animation rendering is turned on via the animation rendering control button, and not when animation rendering is turned off via the animation rendering control button. This embodiment of the application improves the fine-grainedness of rendering control by controlling particle rendering or animation rendering separately.

[0220] In another possible implementation, S808-S813 may not be necessary, in which case all areas in the video frame can be used as rendering areas, thus improving rendering efficiency. In this embodiment, S808-S813 can determine rendering areas that will not be occluded, and rendering in these areas will not obstruct the rendering effect. This reduces the waste of rendering resources and improves the resource utilization rate of rendering resources.

[0221] In another possible implementation, S808-S813 can also be replaced by comparing the depth of the rendered effect with the depth information of each region in the video frame. However, the embodiments of this application improve the efficiency of depth comparison by using a multi-level depth information comparison method through S808-S813.

[0222] It should be noted that the solutions in this application embodiment can also be applied to further processing of the video after the video shooting is completed, and are not limited to processing during the shooting process.

[0223] Please see Figure 11 , Figure 11 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 11 The methods shown may include:

[0224] S1101. Display a first interface. The first interface is used to display images captured by the camera of the electronic device. The first interface includes a first button, and no target animation effects are added to the images displayed on the first interface. The target animation effects include particle rendering and / or animation rendering.

[0225] For example, the first interface can be, for instance, Figure 5 The interface shown in (b) is shown in the diagram. The first button could be, for example, […]. Figure 5 The rendering mode button is shown in (b) above. Particle rendering can be referenced... Figure 5 The rendering shown in (c) can be referenced for animation rendering. Figure 5 The rendering shown in (c) is an example. Again, by way of example, the first interface could be, for example, Figure 6 In the interface shown in (b), the first button can be, for example, Figure 6 The rendering control buttons are shown in (b) above. As another example, the first interface could be, for example, a... Figure 6 In the interface shown in (a), the first button can be, for example, Figure 6 (a) Portrait Mode button.

[0226] S1102. In response to the operation of the first button, a second interface is displayed. The second interface is used to display the image captured by the camera of the electronic device, and the image displayed on the second interface has a target animation effect added.

[0227] The second interface can be, for example, Figure 5 The interface shown in (c) can also be Figure 6 The interface shown in (d) is shown in the image.

[0228] In one possible implementation, the second interface includes a second button, and the method further includes:

[0229] In response to an action on the second button in the second interface, the third interface is displayed, and the image displayed in the third interface does not have the target animation effect added; in response to an action on the second button in the third interface, the fourth interface is displayed, and the image displayed in the fourth interface has the target animation effect added.

[0230] For example, the third interface can be, for instance, Figure 6 In the interface shown in (f), the second button could be, for example, Figure 6 The rendering control button shown in (f) in the fourth interface can be, for example, the fourth interface. Figure 6 The interface shown in (d) is shown in the image.

[0231] In another possible implementation, the buttons for controlling the on or off of the target animation are two different buttons, which can improve the independence of turning the target animation on or off.

[0232] In one possible implementation, the method further includes, before displaying the first interface:

[0233] The fifth screen is displayed, which includes the camera app icon; the first screen is also displayed, including:

[0234] In response to an action on the camera app icon, the first screen is displayed, which is the camera preview screen.

[0235] For example, the fifth interface could be, for instance, Figure 5 The interface shown in (a) is, for example, the first interface. Figure 5 The interface shown in (b) allows the target animation to be hidden when entering the camera preview interface, and then enabled when the user needs it, thus preserving the authenticity of the captured image and improving the user experience.

[0236] Another possible implementation is to display the target animation when entering the camera preview interface, which can improve the visual effect of the displayed image.

[0237] In one possible implementation, the second interface is a video recording interface, which includes a third button. The method also includes:

[0238] In response to the first operation on the third button, video recording is started; in response to the second operation on the third button, video recording ends and the target video is obtained. The target video has a target animation effect added. The target video includes the video recorded by the camera within the target time period, which includes the time period between the response to the first operation and the response to the second operation.

[0239] The target video can be a video obtained through video recording. In this embodiment, after the target animation effect is enabled, the recorded video also has the target animation effect added, which can improve the visual effect of the recorded video.

[0240] In one possible implementation, before displaying the second interface in response to an action on the first button, the method further includes:

[0241] The method further includes: acquiring a first image captured by the camera of an electronic device; extracting information from the first image, the information of which includes scene information and / or object information, the scene information representing the scene of the first image and the object information representing the action of the first object in the first image; acquiring a first mapping relationship, the first mapping relationship representing the mapping relationship between image information and motion effects; and acquiring the target motion effect corresponding to the information of the first image from the first mapping relationship; the method also includes: when displaying a second interface, displaying the first image on the second interface and adding the target motion effect corresponding to the information of the first image to the first image.

[0242] The first image can also be referred to as a video frame. This step can be referred to in the relevant descriptions in S801-S816, and will not be repeated here.

[0243] Another possible implementation is to choose fixed animation effects, which can save the computing resources required for image processing.

[0244] In one possible implementation, the method further includes the following steps before displaying the second interface:

[0245] The depth information of the first image is obtained, including the first depth of any image region in the first image, wherein the first depth of the image region indicates the depth of the image region in the first image; the second depth of the target animation is obtained, wherein the second depth indicates the depth of the target animation in the first image; and the target animation corresponding to the information of the first image is added to at least one image region based on the first depth and the second depth of any image region, wherein for the image region to which the target animation is added, the first depth of the image region is greater than the second depth of the target animation added to the image region.

[0246] In this context, the image region can also be simply referred to as a region. In this embodiment, a single pixel can be considered as an image region, or multiple adjacent pixels can be considered as an image region; no limitation is imposed. This embodiment can be referred to in the relevant descriptions of S808-S814, which will not be repeated here.

[0247] In another possible implementation, target motion effects can be added to any image region, which reduces the system resources required to add motion effects to any particular region.

[0248] In one possible implementation, the depth information of the first image includes L-level depth information, the first depth of an image region in the K-level depth information is related to the first depths of at least two image regions in the (K-1)-level depth information, an image region in the K-level depth information includes at least two image regions in the (K-1)-level depth information, 1 < K ≤ L, and K and L are integers. Based on the first and second depths of any image region, a target animation corresponding to the information of the first image is added to at least one image region, including:

[0249] Obtain the first depth of the image region where the target motion effect is located from the K-th level depth information; compare the first depth obtained from the K-th level depth information with the second depth; if the first depth obtained from the K-th level depth information is greater than the second depth, add the target motion effect to the image region where the target motion effect is located in the K-th level depth information; if the first depth obtained from the K-th level depth information is less than or equal to the second depth, obtain the first depth of the image region where the target motion effect is located from the (K-1)-th level depth information, so that if the first depth obtained from the (K-1)-th level depth information is greater than the second depth, add the target motion effect to the image region where the target motion effect is located in the (K-1)-th level depth information.

[0250] The specific values ​​of L and K can be set as needed and are not restricted here. For example, L can be 4.

[0251] The embodiments of this application can be referred to S808-S813 and... Figure 9 The relevant explanations will not be repeated here.

[0252] Another possible implementation is to omit L-level depth information, which means performing depth comparison in any image region.

[0253] In one possible implementation, the first depth of an image region in the K-th level depth information is the minimum of the first depths of at least two image regions in the (K-1)-th level depth information.

[0254] Please see Figure 12 , Figure 12 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 12 The methods shown may include:

[0255] S1201. Display the first interface. The first interface is used to display the image captured by the camera of the electronic device. The first interface includes a fourth button and a fifth button. The image displayed on the first interface does not have particle rendering or animation rendering.

[0256] This step can be referred to in the description of S1201, and will not be repeated here. The first interface of this embodiment can be, for example, Figure 7 The interface shown in (b) is shown in the image.

[0257] S1202. In response to an operation on the fourth button in the first interface, a sixth interface is displayed. The sixth interface is used to display an image captured by the camera of the electronic device. The sixth interface includes a fifth button. The image displayed on the sixth interface does not have particle rendering but has animation rendering.

[0258] For example, the sixth interface could be, for instance, Figure 7 The interface shown in (c) is as follows. The fourth button could be, for example, an animation control button. The fifth button could be, for example, a particle control button.

[0259] S1203. In response to an operation on the fifth button in the sixth interface, a seventh interface is displayed. The seventh interface is used to display an image captured by the camera of the electronic device. The image displayed in the seventh interface has particle rendering and animation rendering added.

[0260] For example, the sixth interface could be, for instance, Figure 7 The interface shown in (e) is shown in the image.

[0261] In one possible implementation, the seventh interface includes a fourth button and a fifth button. After displaying the seventh interface in response to an action on the fifth button in the sixth interface, the method further includes:

[0262] In response to an action on the fourth button in the seventh interface, an eighth interface is displayed. The eighth interface displays an image captured by the camera of the electronic device. The eighth interface includes a fifth button. The image displayed in the eighth interface has particle rendering added but no animation rendering added.

[0263] In response to an action on the fifth button in the eighth interface, the ninth interface is displayed. The ninth interface displays an image captured by the camera of the electronic device. The image displayed in the ninth interface does not have particle rendering or animation rendering.

[0264] For example, the eighth interface could be, for instance, Figure 7 The interface shown in (g) is, for example, the ninth interface. Figure 7 The interface shown in (i) is shown in the diagram.

[0265] In another possible implementation, the buttons for controlling the animation rendering to start and the buttons for controlling the animation rendering to stop can be different buttons, and / or the buttons for controlling the particle rendering to start and the buttons for controlling the particle rendering to stop can be different buttons.

[0266] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0267] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0268] The image processing method of the present application embodiments has been described above. The apparatus for performing the above method provided in the present application embodiments is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the present application embodiments can perform the steps in the above list sorting method.

[0269] The image processing method provided in this application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can be referred to the above-described related descriptions, which will not be repeated here.

[0270] This application provides a terminal device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the terminal device to perform the above-described method.

[0271] like Figure 13 This is a schematic diagram of a chip structure provided in an embodiment of this application. The chip includes one or more processors 1301, a communication line 1302, a communication interface 1303, and a memory 1304.

[0272] In some implementations, memory 1304 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0273] The methods described in the embodiments of this application can be applied to processor 1301, or implemented by processor 1301. Processor 1301 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit in the hardware of processor 1301 or by instructions in software form. The processor 1301 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. Processor 1301 can implement or execute the various processing-related methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0274] The steps of the method described in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). This storage medium is located in memory 1304, and processor 1301 reads information from memory 1304 and, in conjunction with its hardware, completes the steps of the above method.

[0275] The processor 1301, memory 1304 and communication interface 1303 can communicate with each other via communication line 1302.

[0276] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0277] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0278] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0279] This application provides a computer program product, which includes a computer program that, when run, causes the computer to perform the above-described method.

[0280] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0281] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized by, Applied to electronic devices, the method includes: The first interface is displayed to display images captured by the camera of the electronic device. The first interface includes a first button, and no target animation effects are added to the images displayed on the first interface. The target animation effects include particle rendering and / or animation rendering. In response to the operation of the first button, a second interface is displayed, which is used to display the image captured by the camera of the electronic device, and the target animation effect is added to the image displayed on the second interface; Before displaying the second interface in response to an operation on the first button, the method further includes: Acquire the first image captured by the camera of the electronic device; Extract information from the first image, the information of the first image including scene information and / or object information of the first image, the scene information representing the scene of the first image, and the object information representing the action of the first object in the first image; Obtain a first mapping relationship, which represents the mapping relationship between image information and motion effects; Obtain the target motion effect corresponding to the information of the first image from the first mapping relationship; The method further includes: When the second interface is displayed, the first image is displayed on the second interface, and the target animation effect corresponding to the information of the first image is added to the first image; The depth information of the first image is obtained, and the depth information of the first image includes the first depth of any image region in the first image, wherein the first depth of the image region indicates the depth of the image region in the first image; Obtain a second depth of the target motion effect, the second depth indicating the depth of the target motion effect in the first image; Based on a first depth and a second depth of any of the image regions, the target motion effect corresponding to the information of the first image is added to at least one of the image regions, wherein, for the image region to which the target motion effect is added, the first depth of the image region is greater than the second depth of the target motion effect added to the image region; The depth information of the first image includes L-level depth information. The first depth of an image region in the K-level depth information is related to the first depths of at least two image regions in the (K-1)-level depth information. The image region in the K-level depth information includes the at least two image regions in the (K-1)-level depth information, 1 < K ≤ L, and K and L are integers. The step of adding the target animation corresponding to the information of the first image in at least one of the image regions based on the first depth and the second depth of any of the image regions includes: Obtain the first depth of the image region where the target motion effect is located from the Kth level depth information; Compare the magnitudes of the first depth and the second depth obtained from the Kth level depth information; If the first depth obtained in the Kth level depth information is greater than the second depth, the target motion effect is added to the image region where the target motion effect is located in the Kth level depth information. If the first depth obtained in the Kth level depth information is less than or equal to the second depth, the first depth of the image region where the target motion effect is located is obtained from the (K-1)th level depth information. If the first depth obtained in the (K-1)th level depth information is greater than the second depth, the target motion effect is added to the image region where the target motion effect is located in the (K-1)th level depth information.

2. The method of claim 1, wherein, The second interface includes a second button, and the method further includes: In response to an operation on the second button in the second interface, a third interface is displayed, wherein the target animation effect is not added to the image displayed on the third interface; In response to an operation on the second button in the third interface, a fourth interface is displayed, and the target animation effect is added to the image displayed in the fourth interface.

3. The method according to claim 1 or 2, characterized in that, Before displaying the first interface, the method further includes: The fifth interface is displayed, which includes a camera application icon; The first interface for display includes: In response to an operation on the camera application icon, the first interface is displayed, which is a camera preview interface.

4. The method according to any one of claims 1-3, characterized in that, The second interface is a video recording interface, which includes a third button. The method further includes: In response to the first operation on the third button, video recording is initiated; In response to the second operation on the third button, video recording ends and a target video is obtained. The target video has the target animation added to it. The target video includes the video recorded by the camera within a target time period, and the target time period includes the time period between the response to the first operation and the response to the second operation.

5. The method of claim 1, wherein, The first depth of the image region in the K-th level depth information is the minimum value of the first depths of the at least two image regions in the (K-1)-th level depth information.

6. An image processing method characterized by, Applied to electronic devices, the method includes: The first interface is displayed, which is used to display the image captured by the camera of the electronic device. The first interface includes a fourth button and a fifth button, and the image displayed on the first interface does not have particle rendering or corresponding target motion effects. In response to an operation on the fourth button in the first interface, a sixth interface is displayed, which is used to display an image captured by the camera of the electronic device. The sixth interface includes the fifth button. The image displayed on the sixth interface does not have the particle rendering but has the animation rendering. In response to an operation on the fifth button in the sixth interface, a seventh interface is displayed, which displays an image captured by the camera of the electronic device. The image displayed in the seventh interface has the particle rendering and the animation rendering added. Before displaying the sixth interface in response to an operation on the fourth button in the first interface, the method further includes: Acquire the first image captured by the camera of the electronic device; Extract information from the first image, the information of the first image including scene information and / or object information of the first image, the scene information representing the scene of the first image, and the object information representing the action of the first object in the first image; Obtain a first mapping relationship, which represents the mapping relationship between image information and motion effects; Obtain the target motion effect corresponding to the information of the first image from the first mapping relationship; The method further includes: When the seventh interface is displayed, the first image is displayed on the seventh interface, and the target animation effect corresponding to the information of the first image is added to the first image. The depth information of the first image is obtained, and the depth information of the first image includes the first depth of any image region in the first image, wherein the first depth of the image region indicates the depth of the image region in the first image; Obtain a second depth of the target motion effect, the second depth indicating the depth of the target motion effect in the first image; Based on a first depth and a second depth of any of the image regions, the target motion effect corresponding to the information of the first image is added to at least one of the image regions, wherein, for the image region to which the target motion effect is added, the first depth of the image region is greater than the second depth of the target motion effect added to the image region; The depth information of the first image includes L-level depth information. The first depth of an image region in the K-level depth information is related to the first depths of at least two image regions in the (K-1)-level depth information. The image region in the K-level depth information includes the at least two image regions in the (K-1)-level depth information, 1 < K ≤ L, and K and L are integers. The step of adding the target animation corresponding to the information of the first image in at least one of the image regions based on the first depth and the second depth of any of the image regions includes: Obtain the first depth of the image region where the target motion effect is located from the Kth level depth information; Compare the magnitudes of the first depth and the second depth obtained from the Kth level depth information; If the first depth obtained in the Kth level depth information is greater than the second depth, the target motion effect is added to the image region where the target motion effect is located in the Kth level depth information. If the first depth obtained in the Kth level depth information is less than or equal to the second depth, the first depth of the image region where the target motion effect is located is obtained from the (K-1)th level depth information. If the first depth obtained in the (K-1)th level depth information is greater than the second depth, the target motion effect is added to the image region where the target motion effect is located in the (K-1)th level depth information.

7. An electronic device, comprising: The electronic device includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as claimed in any one of claims 1 to 5, or to perform the method as claimed in claim 6.

8. A chip system, characterized by The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 5, or to perform the method as described in claim 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as claimed in any one of claims 1 to 5, or to perform the method as claimed in claim 6.

10. A computer program product, characterised in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 5, or to perform the method as described in claim 6.

Images