Image processing method and apparatus

Abstract

Embodiments of the present application provide an image processing method and device. The method comprises: obtaining a first image and material information, the first image comprising a first object, the first object in the first image corresponding to a first material, and the material information indicating a second material. Based on the first image and the material information, an image space modal basis of the first object and a second image are obtained, wherein the image space modal basis is used to simulate the dynamics of the first object composed of the second material in the real world, and the first object in the second image corresponds to the second material. In response to a received first user operation, first video data is obtained based on an image control modal basis and the second image. Wherein: the first object in the first video data corresponds to the second material, and the first object moves in the first video data with the first user operation action. Thus, the interaction between the image and the user is realized.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method and apparatus. Background Technology

[0002] With the development of image processing technology, some objects in video data can now interact with users. However, current video data generation methods typically involve pre-generating animations, where objects move along predetermined trajectories. This results in a relatively limited interaction method with the user, negatively impacting the user's visual experience. Summary of the Invention

[0003] This application provides an image processing method and apparatus. In this method, when a first object interacts with a user, it simulates real-world dynamics, thereby enhancing the user's visual experience.

[0004] In a first aspect, this application provides an image processing method. The method includes: acquiring a first image and material information, wherein the first image includes a first object, the first object in the first image corresponds to a first material, and the material information indicates a second material. Based on the first image and material information, acquiring an image spatial modal basis of the first object and a second image, wherein: the image spatial modal basis is used to simulate the dynamics of the first object, which is composed of a second material, in the real world, and the first object in the second image corresponds to the second material. In response to a received first user operation, acquiring first video data based on the image control modal basis and the second image. Wherein: the first object in the first video data corresponds to the second material, and the first object moves with the first user operation in the first video data. Thus, this application generates an image spatial modal basis after material transfer, i.e., an image spatial modal basis corresponding to the second material, to realize the interaction between the first object corresponding to the second material and the user based on this image control modal basis. During the interaction between the first object and the user, the first object's movements simulate the dynamics of the first object made of the second material in the real world, thus providing a more flexible animation generation method. The material of the first object can be changed according to user needs or scene requirements. Moreover, the vibration pattern (or vibration shape) of the first object after the material change during the interaction with the user simulates the vibration pattern of the first object made of that material in the real world. This makes the dynamics of the first object displayed in the video data more realistic, and can simulate different actions according to different user operations, increasing the fun of image interaction and improving the user experience.

[0005] In one possible implementation, the method further includes: responding to a received second user operation, acquiring dynamic image data containing a second object, the second object moving according to a preset trajectory; acquiring second video data based on the dynamic image data of the second object, the image control modal base, and the second image; and displaying the second video data, wherein: the first object in the second video corresponds to the second material, and the first object moves along with the second object. Thus, the first object in this application can also interact with the second object, further enriching the interaction scenarios and improving the user experience.

[0006] In one possible implementation, obtaining the first image and material information includes: in response to a received third user operation, obtaining the first image; and in response to a received fourth user operation, extracting the material information from the third image. In this way, the present application allows the user to specify an image as the first image and to specify the material to be replaced, increasing the freedom of the first object generation scene and enabling the user to customize the generation of corresponding video data.

[0007] For example, the second image and the first video data can be used as wallpaper, incoming call background, chat background, etc.

[0008] In one possible implementation, obtaining the image spatial mode basis of a first object and a second image based on a first image and material information includes: identifying at least one object in the first image; segmenting the at least one object in the first image to obtain image data of the at least one object; and obtaining the image spatial mode basis of the first object based on the image data and material information of the at least one object; wherein the at least one object includes the first object. Thus, by identifying and segmenting objects to obtain independent image data for each object, this application can generate the image spatial mode basis corresponding to the first object in subsequent processes without processing other background images in the image, effectively reducing image processing complexity.

[0009] In one possible implementation, obtaining the image spatial modality basis of a first object based on image data and material information of at least one object includes: using the image data and material information of at least one object as input to an image spatial modality basis generation network, and obtaining the image spatial modality basis of the first object output by the image spatial modality basis generation network; wherein: the image spatial modality basis generation network is trained based on an image database, the image database including multiple training images and material information for indicating different materials, and the training images include the first object. Thus, by pre-training the image spatial modality basis generation network, image control modal bases corresponding to different first objects and different materials can be generated based on the network.

[0010] In one possible implementation, the first material is a non-rigid material. In this way, this application can generate a corresponding simulation video for an image of any object made of a non-rigid material in the real world. For example, it could be a person, an animal, or a plant.

[0011] In one possible implementation, the type of the first user action includes at least one of the following: air gesture type, touch gesture type, and blowing.

[0012] In one possible implementation, in response to a received first user operation, first video data is acquired based on an image control modal basis and a second image, including: determining force information acting on a first object based on the first user operation, the force information including the force direction and magnitude; acquiring optical flow information based on the force information and the image spatial modal basis; wherein: the optical flow information is used to indicate the optical flow of each pixel of the first object; and acquiring the first video data based on the optical flow information and the second image.

[0013] Secondly, this application provides an image processing method applied to a terminal device. The method includes: in response to a received selection operation, acquiring a first image, the first image including a first object, the first object in the first image corresponding to a first material; in response to a received extraction operation, extracting material information from a third image, the material information indicating a second material; acquiring and displaying a second image based on the first image and the material information, wherein: the first object in the second image corresponds to the second material; in response to a received interaction operation on the first object in the second image, generating and displaying first video data; wherein: the first object in the first video data corresponds to the second material, and the first object in the first video data moves with the interaction operation.

[0014] In one possible implementation, the method further includes: displaying the second image after the first video data finishes playing; displaying a second object in the second image in response to a received drag operation, wherein the movement trajectory of the second object is consistent with the drag operation; moving the second object according to a preset trajectory in response to a received raise operation; and generating and displaying the second video data if interaction occurs between the second object and the first object during the movement of the second object according to the preset trajectory, wherein the first object in the second video corresponds to the second material, and the first object moves with the second object.

[0015] In one possible implementation, the type of the interactive operation includes at least one of the following: air gesture type, touch gesture type, and blowing.

[0016] Thirdly, this application provides an image processing apparatus, comprising: an acquisition module for acquiring a first image and material information, wherein the first image includes a first object, the first object in the first image corresponds to a first material, and the material information indicates a second material; the acquisition module for acquiring an image spatial mode basis and a second image of the first object based on the first image and the material information, wherein: the image spatial mode basis is used to simulate the dynamics of the first object composed of the second material in the real world, and the first object in the second image corresponds to the second material; and a simulation module for acquiring first video data based on the image control mode basis and the second image in response to a received first user operation; wherein: the first object in the first video data corresponds to the second material, and the first object moves with the first user operation in the first video data.

[0017] In one possible implementation, the acquisition module is further configured to acquire dynamic image data of the second object in response to a received second user operation; the simulation module is further configured to acquire second video data based on the dynamic image data of the second object, the image control modal base, and the second image, wherein the first object in the second video corresponds to the second material, and the first object moves with the second object in the second video data.

[0018] In one possible implementation, the acquisition module is specifically used to: acquire a first image in response to a received third user operation; acquire a third image in response to a received fourth user operation; and acquire material information based on the third image.

[0019] In one possible implementation, the acquisition module is specifically used for: identifying at least one object in the first image; segmenting the at least one object in the first image to obtain image data of the at least one object; and obtaining the image spatial mode basis of the first object based on the image data and material information of the at least one object; wherein: the at least one object includes the first object.

[0020] In one possible implementation, the acquisition module is specifically used to: take the image data and material information of at least one object as input to the image spatial modality basis generation network, and acquire the image spatial modality basis of the first object output by the image spatial modality basis generation network; wherein: the image spatial modality basis generation network is trained based on an image database, the image database includes multiple training images and material information for indicating different materials, and the training images include the first object.

[0021] In one possible implementation, the first material is a non-rigid material.

[0022] In one possible implementation, the type of the first user action includes at least one of the following: air gesture type, touch gesture type, and blowing.

[0023] In one possible implementation, the simulation module is specifically used for: determining force information acting on a first object based on a first user operation, the force information including the direction and magnitude of the force; acquiring optical flow information based on the force information and the image spatial mode base; wherein: the optical flow information is used to indicate the optical flow of each pixel of the first object; and acquiring first video data based on the optical flow information and the second image.

[0024] Fourthly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory, wherein the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method in the first aspect or any possible implementation thereof.

[0025] Fifthly, embodiments of this application provide a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the method in the first aspect or any possible implementation thereof.

[0026] In a sixth aspect, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a computing device, enable the computing device to perform the method in the first aspect or any possible implementation thereof.

[0027] In a seventh aspect, embodiments of this application provide a terminal device, including a memory and a processor, wherein the processor is configured to execute instructions stored in the memory to cause the terminal device to perform the method in the first aspect or any possible implementation thereof. Attached Figure Description

[0028] Figure 1 A schematic diagram of a communication system as an example;

[0029] Figure 2 A schematic diagram of the hardware structure of a terminal is shown as an example.

[0030] Figure 3 A schematic diagram of the software structure of a terminal as an example;

[0031] Figure 4 This is a schematic flowchart illustrating an exemplary image processing method;

[0032] Figure 5 A schematic diagram of a user interface is shown as an example;

[0033] Figure 6 This is a schematic diagram illustrating a material transfer network as an example.

[0034] Figure 7This is a schematic diagram illustrating the processing flow of a material transfer network as an example.

[0035] Figure 8 This is a schematic diagram of the simulation process as an example.

[0036] Figures 9A to 9D A schematic diagram of a user interface is shown as an example;

[0037] Figure 10 This is a schematic diagram illustrating an image processing flow as an example.

[0038] Figure 11 This is a schematic diagram of the structure of an image processing apparatus as an example.

[0039] Figure 12 This is a schematic diagram illustrating the structure of a computing device as an example.

[0040] Figure 13 This is a schematic diagram illustrating the structure of a computing device as an example.

[0041] Figure 14 This is a schematic diagram illustrating the structure of a computing device cluster as an example. Detailed Implementation

[0042] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0043] Before describing the technical solutions of the embodiments of this application, the communication system of the embodiments of this application will first be described with reference to the accompanying drawings. (Refer to...) Figure 1 , Figure 1 This is a schematic diagram illustrating an exemplary communication system. The communication system includes, but is not limited to, terminals and the cloud. The terminal can be an electronic device with a display function, such as a computer, mobile phone, tablet, or wearable device. It should be noted that in practical applications, the number of terminals can be one or more. Figure 1 The number of terminals in the illustrated communication system is merely an adaptive example. For instance, the cloud can be a cloud server cluster, which includes one or more cloud servers. The terminals and the cloud can interact via a communication connection, which can be the Internet, the World Wide Web, a wide area network, a local area network, etc., and this application does not limit this.

[0044] Figure 2 A structural schematic diagram of terminal 100 is shown. It should be understood that... Figure 2 The terminal 100 shown is only one example of a terminal, and the terminal 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. Figure 2The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0045] Terminal 100 may include: processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0046] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0047] The controller can serve as the central nervous system and command center of the terminal 100. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0048] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0049] The wireless communication function of terminal 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0050] Terminal 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0051] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. In some embodiments, terminal 100 may include one or N displays screens 194, where N is a positive integer greater than 1.

[0052] Terminal 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0053] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0054] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, terminal 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0055] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when terminal 100 selects a frequency point, the DSP can perform Fourier transforms on the frequency energy.

[0056] Video codecs are used to compress or decompress digital video. Terminal 100 may support one or more video codecs. Thus, terminal 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0057] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in terminals, such as image recognition, facial recognition, speech recognition, and text understanding.

[0058] Terminal 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0059] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0060] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal 100 can listen to music or make hands-free calls through the speaker 170A.

[0061] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal 100 receives a phone call or voice message, the receiver 170B can be brought close to the listener's ear to hear the voice.

[0062] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal 100 may have at least one microphone 170C. In some embodiments, terminal 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal 100 may have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0063] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0064] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Terminal 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, terminal 100 detects the intensity of the touch operation based on pressure sensor 180A. Terminal 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0065] The gyroscope sensor 180B can be used to determine the motion attitude of the terminal 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the terminal 100 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal 100's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the terminal 100 through reverse movement, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0066] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0067] The magnetic sensor 180D includes a Hall sensor. The terminal 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the terminal 100 is a flip phone, the terminal 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0068] The 180E accelerometer can detect the magnitude of acceleration of terminal 100 in various directions (typically three axes). When terminal 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the terminal's posture and applied to applications such as screen orientation switching and pedometers.

[0069] A distance sensor 180F is used to measure distance. The terminal 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, the terminal 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0070] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The terminal 100 emits infrared light outward through the LED. The terminal 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal 100. When insufficient reflected light is detected, the terminal 100 can determine that there is no object near the terminal 100. The terminal 100 may use the proximity sensor 180G to detect when a user holds the terminal 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and screen locking.

[0071] The ambient light sensor 180L is used to sense the ambient light intensity. The terminal 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light intensity. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the terminal 100 is in a pocket to prevent accidental touches.

[0072] The fingerprint sensor 180H is used to collect fingerprints. The terminal 100 can use the characteristics of the collected fingerprints to unlock the device, access application locks, take photos with fingerprints, and answer calls with fingerprints.

[0073] Temperature sensor 180J is used to detect temperature. In some embodiments, terminal 100 uses the temperature detected by temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, terminal 100 reduces the performance of the processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, terminal 100 heats battery 142 to prevent abnormal shutdown of terminal 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, terminal 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0074] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of terminal 100, in a different position than display screen 194.

[0075] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0076] The software system of terminal 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered Android system as an example to exemplify the software structure of terminal 100.

[0077] Figure 3 This is a software structure block diagram of the terminal 100 according to an embodiment of this application.

[0078] The layered architecture of Terminal 100 divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into an application layer, a framework layer, and a kernel layer from top to bottom.

[0079] The application layer can include a series of application packages.

[0080] like Figure 3 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0081] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions.

[0082] like Figure 3 As shown, the framework layer may include, but is not limited to: window manager, content provider, simulation module, rendering module, resource manager, view system, etc.

[0083] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0084] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0085] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0086] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0087] The simulation module is used to generate simulation results for the first object. The simulation results include the displacement (i.e., optical flow) of each pixel in the image at each time step. Optionally, gallery applications, video applications, chat applications, wallpaper applications, etc., can call the terminal's simulation and rendering modules to obtain video data. Optionally, system applications such as gallery applications and video applications, or other third-party applications, can also implement the simulation module, or the functions implemented by the simulation and rendering modules, to generate and display the second image and video data containing the first object. For example, a chat application can display the second image as a chat background, and the user can interact with the chat background image so that the first object in the chat background image moves with the interaction. As another example, the images (including the first image and the second image) in this embodiment can also be preview images. For example, during the photo-taking process, the camera application displays a preview image, and the user can interact with the preview image so that the first object in the preview image moves with the interaction.

[0088] The rendering module is used to render based on simulation results and generate video data.

[0089] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0090] Reference Figure 4 , Figure 4 This is a schematic flowchart illustrating an exemplary image processing method, which includes, but is not limited to, the following steps:

[0091] S401, Obtain a first image and material information. The first image includes a first object, the first object in the first image corresponds to a first material, and the material information indicates a second material.

[0092] For example, a terminal (or an application, such as a wallpaper application or a gallery application) acquires a first image. The first image may optionally be a static image, i.e., an image frame. Specifically, the terminal may receive a user operation (e.g., a third user operation), taking the user operation of selecting an image from the gallery as an example. (Refer to...) Figure 5 , Figure 5 This is an illustrative user interface diagram. The user selects a static image (or picture) 501 from the gallery. In response to the received selection operation, the terminal retrieves the image 501 selected by the user (i.e., the first image) from the gallery.

[0093] For example, the first image includes a first object. In this embodiment, the material of the first object (e.g., the first material described in this embodiment) may optionally be a non-rigid material. For example: people, flowers, animals, dolls, etc. It can be understood that some non-rigid material objects in this embodiment can vibrate in response to forces in reality. For example, taking the rose in image 501 as an example, in the real world, the rose, as a non-rigid material object, can vibrate (or move) in response to forces applied to it (which can be forces applied by people or forces applied by natural wind, etc.). As another example, if the object is a rigid material, such as a stone, it cannot vibrate in response to forces applied to it (referring to vibrations visible to the naked eye). It can be understood that the simulation interaction method in this embodiment simulates the vibration mode of the first object in the real world (e.g., including vibration frequency and vibration amplitude, etc.) to generate corresponding video data, that is, the vibration mode of the first object displayed in the video data is similar to that in the real world. However, if the first object is a rigid body, its vibration mode in the real world is generally unavailable, making it impossible to simulate.

[0094] For example, the terminal acquires material information. Specifically, the user can select an image from a gallery (or other applications or image databases, such as alternative reference images provided in the cloud). In response to the received selection operation (e.g., a fourth user operation), the terminal acquires an image (referred to here as a reference image, i.e., the third image described in this application, to distinguish it from the first image). The reference image is used to provide material information. For example, if the user wants to change the material of the flower in the first image to jelly, the user can select an image containing an object with a jelly-like material (e.g., jelly) as the reference image.

[0095] Optionally, the user can also select the material to be extracted from the reference image; for example, the user can click or circle the object from which the material needs to be extracted (e.g., jelly). In response to the received extraction operation, the terminal can extract material information based on the reference image. Specifically, after acquiring the reference image, the terminal can input the reference image into the material transfer network. The material transfer network processes it (e.g., material extraction processing) to obtain the corresponding material information.

[0096] Reference Figure 6 , Figure 6 This is a schematic diagram illustrating an exemplary material transfer network. A material transfer network (also known as an image-space modal bases material transfer network) includes, but is not limited to: a material extraction module (or material extraction network), a content recognition module (or content recognition network), a segmentation module (or segmentation network), and a generation module (or an image-space modal bases generation network).

[0097] A material extraction module is used to extract material information. For example, the material extraction module can extract corresponding material information based on an acquired image (e.g., a reference image). The material information characterizes the material of the image, where the material can be understood as the visual and physical properties of an object's surface. Rendering the image based on the material information can create realistic rendering effects. This application extracts material information and maps 2D images onto the surface of a 3D model based on this information, enabling the model to present richer visual details. For example, given a reference image containing jelly, the material extraction module (which may be a material extraction network) can extract corresponding material information, which characterizes the jelly's material. Optionally, the material information includes, but is not limited to, at least one of the following: texture and lighting features, etc.

[0098] The content recognition module performs content recognition on the image to obtain content recognition results, which include one or more objects. For example, the content recognition network could be a Grounding-DINO network. For instance, given an image containing a person, a cat, and a dog, the content recognition network can identify the person, cat, and dog in the image. The content recognition network can be trained on a content recognition database containing multiple images, each of which may include different objects.

[0099] The segmentation module is used to segment one or more objects contained in the content recognition result (also known as image segmentation processing) to obtain the image segmentation result. Optionally, the segmentation network can be a SAM network. For example, given an input image containing a cat, the content recognition network can identify the object in the image as a cat. The segmentation module can segment the image of the cat, which can be understood as segmenting along the outline of the cat's image, resulting in an image of the cat that does not include other background images.

[0100] The generation module is used to generate image-space modal bases, hereinafter referred to as modal bases. Image-space modal bases are the structural representations of objects in image space, derived from modal analysis of object deformation in image space, and serve as the basis for dynamic simulation of objects in image space. Image space refers to the analysis and simulation performed at the image level, without 3D or 2D meshes or other physical models.

[0101] Please refer to Figure 6 Specifically, the terminal inputs image 501 (i.e., the first image) and reference image 502 into the material transfer network. The material extraction module in the material transfer network performs material extraction processing on the reference image 502 to obtain material information. Optionally, the material information is used to characterize the material of an object in the image. For example, if the object in image 502 is jelly, the material information obtained by the material extraction module is used to indicate that the material is jelly. Material information includes, but is not limited to, texture and lighting features.

[0102] In one possible implementation, the terminal could also provide a selection of alternative materials for the user to choose from. Examples include jelly-like materials, metallic materials, and knitted materials. The user can select the corresponding material through a selection interface provided by the terminal, which can pre-store material information for each type of material.

[0103] In another possible implementation, the terminal can also provide several alternative reference images for the user to choose from. These could include images containing jelly or plush toys, and the terminal can pre-store the material information of the corresponding alternative reference images. The user can select the corresponding reference image through the terminal's selection interface, and the terminal, upon receiving the selected reference image, can retrieve the corresponding material information from the pre-stored material information.

[0104] S402, based on the first image and material information, obtain the image space mode basis and the second image of the first object, wherein: the image space mode basis is used to simulate the dynamics of the first object made of the second material in the real world, and the first object in the second image corresponds to the second material.

[0105] For example, the terminal can obtain a second image based on a first image and material information. Specifically, the terminal can replace the material of at least one object in the first image, including a first object, with a second material to obtain the second image. That is, the material of at least one object in the second image is the second material. For example, the flower in image 501 in the second image can have a jelly material (i.e., the second material). Specifically, the terminal can input the first image and material information into an image material transfer network to obtain a second image output by the image material transfer network, wherein the flower in the second image is visually displayed as a jelly material. The image material transfer network can be trained based on images containing different objects and different material information.

[0106] Optionally, the user can also choose to perform style transfer or background replacement on the background style of the first image. In response to the received user operation, the terminal obtains the background portion of the second image with the style or content specified by the user. For example, if the background of the first image is a blue sky, the terminal, in response to the received user operation, obtains the second image, in which the flowers are displayed with a jelly-like texture, and the background of the second image is a desert image.

[0107] In this embodiment, the object that can generate modal bases can also be called an interactive object; that is, the first object in this embodiment is an interactive object. The first image may include multiple objects, which may include interactive objects and non-interactive objects (e.g., objects made of rigid materials, such as stones).

[0108] In one example, the first image may include multiple first objects, and the terminal may replace the material of each first object in the first image with a second material.

[0109] In another example, the terminal can replace the material of a specified first object (e.g., the flower in image 501) with a second material based on a user-selected action.

[0110] In this embodiment, the example of the terminal converting a user-specified first object into a second material is used for illustration. For instance, in the second image, the material of the flower becomes a jelly material, while the materials of other objects (not shown in the figure) remain the same as in the first image.

[0111] In this embodiment, the terminal obtains the image space modal bases (hereinafter referred to as modal bases) of the first object based on the first image and material information. The modal bases of the first object are used to simulate the dynamics (also referred to as vibration states or vibration modes) of the first object composed of the second material in the real world.

[0112] For details, please refer to Figure 7 The content recognition module performs content recognition on image 501 (i.e., the first image) and obtains the content recognition result. Content recognition is used to identify at least one object in image 501. For example, this may include object 701. This example only uses one object as an example; as mentioned above, multiple objects can be identified in an image. In this embodiment, image 501 includes at least one interactive object, and may also include at least one non-interactive object. Images that do not include interactive objects are not processed in this application.

[0113] The segmentation module can segment objects based on content recognition results (also known as image segmentation) to obtain image segmentation results. Image segmentation is used to separate objects; for example, the image segmentation result includes image data 702 of a flower.

[0114] The generation module can obtain the image spatial modal base 703 corresponding to the first object of the second material based on the material information and image segmentation results (i.e., the flower image data 702). In this embodiment, the segmented object includes image data of at least one interactive object, and may also include image data of some non-interactive objects. The generation network can generate modal bases corresponding to the interactive objects of the second material, but does not output modal bases corresponding to the non-interactive objects.

[0115] In essence, modal bases can generally simulate the vibration patterns of a first object in the real world. In this embodiment, the generative network, based on material information, can generate modal bases corresponding to the first object after material transfer, thus simulating the vibration patterns of the first object corresponding to the second material in the real world. For example, in a scenario without material transfer, the generative network can generate modal bases corresponding to the first object (e.g., a flower). These modal bases can be used to simulate the vibration patterns of the flower in the real world, so that during the interaction between the flower and the user, the flower displayed in the video data can simulate the vibration patterns of the flower in the real world. In this embodiment, the material transfer network performs material transfer on the first object (i.e., the flower) to generate modal bases corresponding to the second material. These modal bases simulate the vibration patterns of a flower with the second material in the real world, such as the vibration patterns of a flower made of jelly material. Thus, during the interaction, the jelly-material flower displayed in the video data can simulate the vibration patterns of a jelly-material flower in the real world.

[0116] In this embodiment, each module (or network) in the material transfer network is trained in the cloud and transmitted to the terminal. Each module (also referred to as a sub-network or sub-module) can be trained individually or in combination.

[0117] Optionally, the public cloud AI infrastructure development platform is a PaaS cloud service within the cloud platform. It's a software platform that leverages the extensive infrastructure resources and software capabilities of the public cloud service provider to assist users (also known as tenants, AI developers, etc.) in building, training, and deploying AI models, as well as developing and deploying AI applications. The interaction between tenants and the AI ​​infrastructure development platform primarily involves: tenants logging into the cloud platform via a client webpage, selecting and purchasing cloud services from the AI ​​infrastructure development platform, and then using the platform's functionalities to conduct full-process AI development.

[0118] In this embodiment, the tenant can train a network (e.g., a general network) on a cloud platform to obtain a material transfer network. Specifically, the training of a generative network is used as an example. In this embodiment, the generative network is trained based on an image database, which includes multiple training images and material information indicating different materials. The training images include a first object (e.g., a flower). During training, modal bases generated from video data of the first object with different materials are used as labels to correct the output of the generative network. This process is repeated until a preset number of training iterations is reached or the loss between the modal bases generated by the generative network and the modal bases generated from the video data reaches a threshold (which can be set according to actual needs). The video data of the first object with different materials can be video data rendered by a model to simulate the dynamics of the first object with different materials in the real world. Alternatively, it can be actual video data. For example, first objects with different materials can be created artificially, such as creating a flower with a jelly-like material and filming its vibration patterns in the real world to obtain corresponding video data. Generating modal bases based on video data can be understood as performing a Fourier transform on the temporal variables of the video pixels (which can also be understood as optical flow, i.e., the movement of pixels between different image frames) to obtain the corresponding frequency domain variables, which can be used to characterize the horizontal and vertical movement of the corresponding pixels at different frequencies. Specific transformation methods can be found in existing technologies and will not be elaborated upon here. Thus, by training the generative network, it can generate modal bases for the first object after material transfer (i.e., the first object corresponding to the second material) based on the input image and material information, for example... Figure 7The modalbases703 in this application can be understood as follows: when a first object corresponds to different materials, its vibration mode in a real scene is different. For example, a real flower and a flower made of plastic will produce different vibration modes under the same force. Therefore, in this embodiment, based on the modal bases after material transfer, corresponding video data is generated, so that when the first object in the image, which has been replaced with a second material, interacts with the user, its vibration mode simulates the vibration mode (or vibration shape, vibration form, dynamic mode, etc.) of the first object made of the second material in the real world.

[0119] For example, the cloud can send the trained material transfer network to the terminal. Optionally, the cloud can update the material transfer network in real time and send the updated material transfer network to the terminal.

[0120] S403, in response to the received first user operation, first video data is obtained based on the image control modal base and the second image; wherein: the first object in the first video data corresponds to the second material, and the first object moves with the first user operation in the first video data.

[0121] For example, after the terminal obtains the modal bases of the first object, it can receive user operations (e.g., the first operation), also known as an interactive operation, and generate corresponding first video data based on the modal bases of the first object and the second image.

[0122] Reference Figure 8 , Figure 8 This is a schematic diagram illustrating the simulation process. Specifically, the terminal performs image material transfer on the first image (i.e., graphic 501) to obtain a second image (i.e., image 801), which is then displayed on the display interface. The first object in image 801 is visually displayed as a jelly material (i.e., the second material). The terminal receives user operations (also known as interactive operations) and obtains the corresponding force information. Based on the force information (including the magnitude and direction of the force), the second image, and the modal bases 703 of the first object, the terminal obtains corresponding video data. In this video data, the first object is a jelly material, and it moves in response to user operations. Its movement pattern in the video data simulates the dynamics (or vibration mode, vibration form, vibration shape, etc.) of the first object (e.g., a flower) made of jelly material in the real world.

[0123] For example, Figures 9A to 9D This is an example of an interactive diagram. In this example, as shown... Figure 9AAs shown, the terminal displays a second image. In this example, the second image is used as the wallpaper for illustration. Figure 9B As shown, the user can slide on the image of the first object to apply force to it. The terminal can generate and display corresponding video data based on the acquired user actions, the second image, and the modal bases of the first object. Figure 9C As shown, in terms of visual effect, the first object (i.e., the flower (including petals, leaves and stem)) displayed as a jelly material moves (or vibrates) with the user's operation, and its movement simulates the vibration pattern of the real world.

[0124] Optionally, the interactive operations in this application embodiment may include, but are not limited to, touch gesture operations, air gesture operations, and blowing air. Touch gesture operations may optionally involve the user operating on a first object displayed on the touchscreen screen via touch, such as tapping or dragging. Air gesture operations may optionally involve the user operating the first object remotely (i.e., without touching the screen), and the gestures may also include dragging or pinching.

[0125] In response to a received operation, the terminal obtains force information acting on the interactive object. This force information includes, but is not limited to, the magnitude and direction of the force. Optionally, for air gestures, the magnitude of the force can be determined based on the speed of the air gesture movement. Of course, in some instances, only the direction of the force can be obtained, while the magnitude of the force is a preset value.

[0126] Optionally, the terminal can capture images via a camera to obtain corresponding air gestures. Alternatively, the terminal can obtain corresponding touch gestures via a touch sensor. Or, the terminal can obtain the amount of air blown through a sensor on the charging port, thereby applying a continuous force to the first object to simulate wind, causing the image displayed in the video data to move (or vibrate) as the user blows. Of course, in other embodiments, the user can also interact with the first object through other devices, such as mouse dragging, keyboard input, or gamepad input. Optionally, the user can also interact with the first object through other operation methods, such as shaking the terminal. The terminal can obtain the corresponding pose information and the magnitude of the shaking force through sensors, causing the first object in the video data to move accordingly.

[0127] In one possible implementation, the user can also select target audio so that the first object vibrates in accordance with the frequency of the audio. The direction of vibration can be random. The amplitude of vibration can be random, or it can vibrate in accordance with the frequency of the audio.

[0128] Specifically, taking touch gesture operation as an example, the terminal can obtain the direction and pressure of the touch gesture through relevant sensors to determine the magnitude and direction of the force f(t) acting on the first object. The terminal can then calculate the optical flow information based on the following formula, which indicates the optical flow (u(t)) of each pixel corresponding to the first object in each image frame:

[0129]

[0130] Where: C=αM+βK,

[0131] diag(m i )=φ T Mφ, φ=[φ1、φ2…φ n ]

[0132]

[0133] Where M, C, and K are the mass matrix, damping matrix, and stiffness matrix, respectively. ω i It is the natural frequency, φ i It is the vibration mode at each natural frequency.

[0134] Based on the above simulation process, the terminal obtains the modalbases(φ) of the first object corresponding to the second material. i Given the optical flow (i.e., the displacement u(t) of each point in the image at each subsequent moment, along with the user-inputted external force f(t), the terminal can calculate the optical flow (i.e., the displacement u(t)) of each point in the image at any subsequent moment. The terminal uses the second image as the first image frame in the video data, and can obtain the optical flow of each pixel of the first object in any image frame of the video data, corresponding to the optical flow in the second image. The rendering module can then draw and render based on the optical flow information and the second image, generating the corresponding video data (i.e., the first video data).

[0135] Optionally, during video generation, due to the movement of pixels in the first object, "holes" may appear in the image, meaning that existing pixels have moved to another position, leaving no pixels at the original location. The rendering module can perform image processing to fill these "holes".

[0136] In one possible implementation, after the video data playback ends, the terminal's display shows the second image. This can be understood as follows: following real-world vibration patterns, the flower vibrates from a large amplitude to a small amplitude until it comes to rest as the user operates it; the image corresponding to its stillness is the second image.

[0137] In one possible implementation, the first object corresponding to the second material can also perform simulated interaction with the second object (also referred to as the simulation object). Specifically, in response to a received user operation (e.g., a second user operation), the terminal acquires the dynamic image data of the second object.

[0138] In one example, the motion image data of the second object could be pre-selected animated video data. For instance, the terminal's display interface shows the second image (e.g., the second image displayed after the first object has come to a standstill, as described above). In the display interface of the second image (e.g.) Figure 9A As shown, a second object option (not shown in the figure) can be displayed, such as: raindrop simulation option, doll simulation option, etc.

[0139] Users can click on the corresponding options to select a specified second object. For example, if a user clicks the raindrop simulation option (which is the corresponding second user operation), the terminal responds to the received user operation by acquiring pre-stored dynamic image data of a raindrop (i.e., the second object). The dynamic image data of the raindrop shows the raindrop falling from top to bottom, meaning its trajectory is from top to bottom.

[0140] For example, the terminal can acquire second video data based on the dynamic image data of the second object, the modal bases of the image control, and the second image. The first object in the second video corresponds to the second material, and the first object moves with the second object in the second video data. Specifically, the terminal acquires the dynamic image data of the second object and can determine the motion state of the second object (including motion direction, speed, acceleration, etc.) based on the dynamic image data. Based on the motion state of the second object, the terminal can determine the force (including direction and magnitude) acting on the first object when the second object touches (i.e., interacts) with the first object. Correspondingly, the terminal can draw and render the vibration pattern during the interaction between the first and second objects based on the force acting on the first object, the modal bases of the first object, and the second image, so that the first object can move with the second object in the displayed video data.

[0141] The above description only uses the user's action of clicking an option as an example. In other embodiments, the user action may be of other types or in other ways.

[0142] For example: Figure 9DAs shown, if the user clicks the "Doll Simulation" option, the terminal can acquire an image of the doll and display it in the second image. This image can be either static or animated. The doll's movements in the animated image are also preset actions. The user can drag the second object, which moves within the second image according to the user's dragging action; its movement trajectory is the dragging trajectory. During dragging, the second object can also incorporate some fun actions, such as shaking or jumping.

[0143] The user moves the second object to the first position (e.g.) Figure 9D After (as shown), the user can raise their hand. In response to the received hand-raising operation, the terminal displays a preset dynamic image of the second object in the second image, and the second object moves according to a preset trajectory. For example, the preset trajectory of the second object is to fall from top to bottom.

[0144] In this example, if the second object interacts with the first object (e.g., colliding) while moving along a preset trajectory, the terminal can generate and display second video data based on the modalbases of the first object and the second image. During the playback of the second video, the first object, made of the second material, moves along with the second object. That is, from the user's perspective, when the second object falls and collides with the first object, the first object vibrates due to the collision.

[0145] Of course, in other embodiments, the user can drag the second object with the mouse and release it at any position, causing it to fall on the screen. The animation of the second object falling as the user drags it is all pre-set animated video data.

[0146] In one possible implementation, the terminal can also trigger the second object display function through other means. Optionally, the triggering methods include, but are not limited to, incoming call reminders, message reminders, etc. For example, taking the second image as wallpaper, the terminal can pre-store dynamic video data of the second object corresponding to the incoming call reminder. For example, when the terminal receives an incoming call reminder, it will display dynamic video data of a doll jumping on the screen. In this embodiment, when the terminal receives an incoming call reminder, it can generate and display second video data based on the dynamic data of the doll jumping, the modal bases of the first object corresponding to the second material, and the second image. During the display of the second video data, the doll in the second video data jumps continuously in the display interface. When the doll touches the first object (e.g., a flower), the flower, displayed as a jelly material, vibrates accordingly.

[0147] In another possible implementation, the terminal can also acquire pre-stored dynamic weather video data based on the current weather conditions. The corresponding second object can include raindrop particles, snow particles, etc. The terminal can acquire the corresponding dynamic weather video data based on the current weather, and generate and display second video data based on the dynamic weather video data, the modal bases of the first object, and the second image. In the second video data, the second object (e.g., raindrop particles) can interact with flowers displayed as a jelly-like material.

[0148] The embodiments described above use the terminal executing S401 to S403 as an example. In another possible implementation, Figure 4 Some steps in the process can also be performed in the cloud. (See reference...) Figure 10 , Figure 10 This is an illustrative diagram of an image processing workflow, which includes, but is not limited to, the following steps:

[0149] S1001 retrieves the first image and material information from the cloud.

[0150] For example, the cloud can obtain the first image and material information from the terminal. The terminal can respond to a received user operation, obtain the first image and material information, and send the first image and material information to the cloud.

[0151] In one possible implementation, the terminal can also send a first image and a reference image to the cloud, and the cloud can obtain material information based on the reference image.

[0152] In another possible implementation, the first image and reference image (or material information) can also be stored in the cloud. The user can select the corresponding first image and reference image (or specify a material) through the cloud platform user interface displayed on the terminal. In response to the received user operation, the terminal sends corresponding user operation information to the cloud to instruct the user to select the first image and reference image. The cloud can then retrieve the first image and reference image (or material information) from cloud storage based on the received user operation information.

[0153] S1002, Obtain the image spatial mode basis of the first object and the second image from the cloud.

[0154] For specific implementation details, please refer to S402; they will not be elaborated here.

[0155] S1003, acquires first video data from the cloud.

[0156] For example, the terminal receives a user operation. The terminal sends an instruction to the cloud, indicating the corresponding user operation. The cloud, in response to the instruction, can obtain the first video data based on the user operation. For specific implementation details, please refer to 403; further explanation is omitted here.

[0157] In one possible implementation, S1003 can also be executed by the terminal. Specifically, the cloud can send the image spatial mode base of the first object and the second image to the terminal, and the terminal can perform the same steps as in S403. Optionally, in this example, the step of acquiring the second image can also be executed by the terminal.

[0158] Reference Figure 11 , Figure 11 This is a schematic diagram of the structure of an exemplary image processing apparatus. This application provides an image processing apparatus, including an acquisition module for acquiring a first image and material information, wherein the first image includes a first object, the first object in the first image corresponds to a first material, and the material information indicates a second material; the acquisition module is used to acquire an image spatial mode basis and a second image of the first object based on the first image and the material information, wherein: the image spatial mode basis is used to simulate the dynamics of the first object composed of the second material in the real world, and the first object in the second image corresponds to the second material; a simulation module is used to acquire first video data based on the image control mode basis and the second image in response to a received first user operation; wherein: the first object in the first video data corresponds to the second material, and the first object moves with the first user operation in the first video data.

[0159] Both the acquisition module and the simulation module can be implemented in software or hardware. For example, the implementation of the first acquisition module will be described below. Similarly, the implementation methods of other modules can refer to the implementation method of the first acquisition module.

[0160] As an example of a software functional unit, a module can include code running on a computing instance. A computing instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0161] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0162] As an example of a hardware functional unit, an acquisition module may include at least one computing device, such as a server. Alternatively, an acquisition module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0163] The acquisition module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the acquisition module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the acquisition module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0164] It should be noted that, in other embodiments, the acquisition module can be used to execute any step in the image processing method, and the simulation module can be used to execute any step in the image processing method. The steps implemented by the acquisition module and the simulation module can be specified as needed, and the image processing device can achieve all its functions by implementing different steps in the image processing method through the acquisition module and the simulation module respectively.

[0165] This application also provides a computing device 1200. For example... Figure 12As shown, the computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1209. The processor 1204, the memory 1206, and the communication interface 1209 communicate with each other via the bus 1202. The computing device 1200 can be a cloud server or a vehicle. It should be understood that this application does not limit the number of processors and memories in the computing device 1200.

[0166] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1202 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1202 may include a path for transmitting information between various components of the computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1209).

[0167] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0168] The memory 1206 may include volatile memory, such as random access memory (RAM). The memory 1206 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0169] The memory 1206 stores executable program code, and the processor 1204 executes the executable program code to implement the functions of the aforementioned acquisition module and simulation module, thereby realizing the image processing method in this embodiment. That is, the memory 1206 stores instructions for executing the image processing method.

[0170] The communication interface 1209 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0171] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device.

[0172] like Figure 13 As shown, the computing device cluster includes at least one computing device 1300. The memory 1306 of one or more computing devices 1300 in the computing device cluster may store the same instructions for executing image processing methods.

[0173] The computing device 1300 includes a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. The processor 1304, the memory 1306, and the communication interface 1308 communicate with each other via the bus 1302.

[0174] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the computing device cluster may also store partial instructions for executing the image processing method. In other words, a combination of one or more computing devices 1300 can jointly execute the instructions for executing the image processing method.

[0175] It should be noted that the memory 1306 in different computing devices 1300 within the computing device cluster can store different instructions, which are used to execute parts of the functions of the aforementioned acquisition module and simulation module. For example, the instructions stored in the memory 1306 of different computing devices 1300 can implement one or more functions of the acquisition module and simulation module included in the aforementioned image processing apparatus.

[0176] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 14 One possible implementation is shown. For example... Figure 14 As shown, the two computing devices 1400A and 1400B are connected via a network. Specifically, they are connected to the network through the communication interface 1408 in each computing device.

[0177] The computing device 1400A includes a bus 1402, a processor 1404, a memory 1406, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate with each other via the bus 1402.

[0178] The computing device 1400B includes a bus 1402, a processor 1404, a memory 1406, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate with each other via the bus 1402.

[0179] In this type of possible implementation, the memory 1406 in computing device 1400A stores instructions for performing the functions of the aforementioned acquisition module. Meanwhile, the memory 1406 in computing device 1400B stores instructions for performing the functions of the aforementioned simulation module.

[0180] It should be understood that Figure 14 The functions of computing device 1400A shown can also be performed by multiple computing devices 1400. Similarly, the functions of computing device 1400B can also be performed by multiple computing devices 1400.

[0181] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 14 and Figure 14 The connection method of the computing device cluster. The difference is that the memory 1406 of one or more computing devices 1400 in the computing device cluster can store the same instructions for executing image processing methods.

[0182] In some possible implementations, the memory 1506 of one or more computing devices 1500 in the computing device cluster may also store partial instructions for executing the image processing method. In other words, a combination of one or more computing devices 1500 can jointly execute the instructions for executing the image processing method.

[0183] It should be noted that the memory 1506 in different computing devices 1500 within the computing device cluster can store different instructions for executing some functions of the aforementioned image processing device. For example, the instructions stored in the memory 1506 of different computing devices 1500 can implement the functions of the acquisition module and simulation module of the aforementioned image processing device.

[0184] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the image processing method described above.

[0185] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the image processing method described in the above embodiments.

[0186] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0187] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0188] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0189] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0190] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Acquire a first image and material information, wherein the first image includes a first object, the first object in the first image corresponds to a first material, and the material information indicates a second material; Based on the first image and the material information, an image space mode basis and a second image of the first object are obtained, wherein: the image space mode basis is used to simulate the dynamics of the first object made of the second material in the real world, and the first object in the second image corresponds to the second material; In response to a received first user operation, first video data is obtained based on the image control modal base and the second image; wherein: the first object in the first video data corresponds to the second material, and the first object moves with the first user operation in the first video data.

2. The method according to claim 1, characterized in that, The method further includes: In response to a received second user operation, dynamic image data containing a second object is acquired, and the second object moves according to a preset trajectory; Based on the dynamic image data of the second object, the modal base of the image control, and the second image, second video data is obtained; The second video data is displayed, wherein: the first object in the second video corresponds to the second material, and the first object moves with the second object.

3. The method according to claim 1, characterized in that, The acquisition of the first image and material information includes: In response to a received third user action, the first image is acquired; In response to the received fourth user action, the material information is extracted from the third image.

4. The method according to claim 1, characterized in that, The step of obtaining the image spatial mode basis and the second image of the first object based on the first image and the material information includes: Identify at least one object in the first image; Segment the at least one object in the first image to obtain image data of the at least one object; Based on the image data of the at least one object and the material information, the image spatial mode basis of the first object is obtained; wherein: the at least one object includes the first object.

5. The method according to claim 4, characterized in that, The step of obtaining the image spatial mode basis of the first object based on the image data of the at least one object and the material information includes: The image data of the at least one object and the material information are used as inputs to an image spatial modality basis generation network to obtain the image spatial modality basis of the first object output by the image spatial modality basis generation network; wherein: the image spatial modality basis generation network is trained based on an image database, the image database includes multiple training images and material information for indicating different materials, and the training images include the first object.

6. The method according to any one of claims 1 to 5, characterized in that, The first material is a non-rigid material.

7. The method according to any one of claims 1 to 6, characterized in that, The type of the first user operation includes at least one of the following: Air gesture types, touch gesture types, and blowing gestures.

8. The method according to any one of claims 1 to 7, characterized in that, The step of responding to a received first user operation by acquiring first video data based on the image control modal base and the second image includes: Based on the first user operation, the force information acting on the first object is determined, and the force information includes the force direction and the force magnitude. Based on the force information and the image spatial mode base, optical flow information is obtained; wherein: the optical flow information is used to indicate the optical flow of each pixel of the first object; The first video data is obtained based on the optical flow information and the second image.

9. An image processing method, characterized in that, Applied to a terminal device, the method includes: In response to a received selection operation, a first image is acquired, the first image including a first object, the first object in the first image corresponding to a first material; In response to the received extraction operation, material information is extracted from the third image, the material information indicating the second material; Based on the first image and the material information, a second image is obtained and displayed, wherein the first object in the second image corresponds to the second material; In response to a received interactive operation on the first object in the second image, first video data is generated and displayed; wherein: the first object in the first video data corresponds to the second material, and the first object in the first video data moves with the interactive operation.

10. The method according to claim 9, characterized in that, The method further includes: After the first video data finishes playing, the second image is displayed; In response to the received drag operation, a second object is displayed in the second image, and the movement trajectory of the second object is consistent with the drag operation; In response to the received hand-raising operation, the second object moves according to a preset trajectory; During the process of the second object moving according to a preset trajectory, if the second object interacts with the first object, second video data is generated and displayed; wherein: the first object in the second video corresponds to the second material, and the first object moves with the second object.

11. The method according to claim 9, characterized in that, The interactive operation includes at least one of the following: Air gestures, touch gestures, and blowing air.

12. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire a first image and material information, wherein the first image includes a first object, the first object in the first image corresponds to a first material, and the material information indicates a second material; The acquisition module is used to acquire the image space mode basis and the second image of the first object based on the first image and the material information, wherein: the image space mode basis is used to simulate the dynamics of the first object made of the second material in the real world, and the first object in the second image corresponds to the second material; The simulation module is used to respond to a received first user operation and acquire first video data based on the image control modal base and the second image; wherein: the first object in the first video data corresponds to the second material, and the first object moves with the first user operation in the first video data.

13. The apparatus according to claim 12, characterized in that, The acquisition module is further configured to, in response to a received second user operation, acquire dynamic image data containing a second object, wherein the second object moves according to a preset trajectory; The simulation module is also used to acquire second video data based on the dynamic image data of the second object, the image control modal base, and the second image; A display module is used to display the second video data, wherein: the first object in the second video corresponds to the second material, and the first object moves with the second object.

14. The apparatus according to claim 12, characterized in that, The acquisition module is specifically used for: In response to a received third user action, the first image is acquired; In response to the received fourth user action, the material information is extracted from the third image.

15. The apparatus according to claim 12, characterized in that, The acquisition module is specifically used for: Identify at least one object in the first image; Segment the at least one object in the first image to obtain image data of the at least one object; Based on the image data of the at least one object and the material information, the image spatial mode basis of the first object is obtained; wherein: the at least one object includes the first object.

16. The apparatus according to claim 15, characterized in that, The acquisition module is specifically used for: The image data of the at least one object and the material information are used as inputs to an image spatial modality basis generation network to obtain the image spatial modality basis of the first object output by the image spatial modality basis generation network; wherein: the image spatial modality basis generation network is trained based on an image database, the image database includes multiple training images and material information for indicating different materials, and the training images include the first object.

17. The apparatus according to any one of claims 12 to 16, characterized in that, The first material is a non-rigid material.

18. The apparatus according to any one of claims 12 to 17, characterized in that, The type of the first user operation includes at least one of the following: Air gesture types, touch gesture types, and blowing gestures.

19. The apparatus according to any one of claims 12 to 18, characterized in that, The simulation module is specifically used for: Based on the first user operation, the force information acting on the first object is determined, and the force information includes the force direction and the force magnitude. Based on the force information and the image spatial mode base, optical flow information is obtained; wherein: the optical flow information is used to indicate the optical flow of each pixel of the first object; The first video data is obtained based on the optical flow information and the second image.

20. A terminal device, characterized in that, Including processor and memory; The processor is configured to execute instructions stored in the memory to cause the terminal device to perform the method as described in any one of claims 1 to 11.

21. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the method as described in any one of claims 1 to 11.

22. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 11.

Images