Video special effect adding method, device and terminal equipment

Abstract

The embodiment of the application discloses a video special effect adding method and device and a terminal equipment, gesture actions are used to replace the original key clicking mode to start the special effect, the complexity of user operation can be reduced, and the special effect can be quickly started when a video call call comes in. According to the area of the region slid by the user's finger or the screen proportion, regional portrait segmentation, background replacement or virtualization can be performed, and the interest, playability and interactivity of the product are improved. Before the portrait segmentation is performed, the original image can be down-sampled, so that the calculation and power consumption of the portrait segmentation are greatly reduced, so that the deep learning portrait segmentation model algorithm requiring large calculation can be run on the mobile terminal with limited computing power and power consumption, and the frame rate requirement of video playing is met.

Description

[0001] This application is a divisional application of Chinese patent application filed on September 25, 2020, with application number 2020110213300 and title "Method, Apparatus and Terminal Equipment for Adding Video Special Effects". The entire contents of this application are included in the parent application. [Technical Field]

[0002] This application relates to the field of mobile communication technology, and in particular to a method, apparatus and terminal device for adding video special effects. [Background Technology]

[0003] Many video calling products now support adding beauty effects, filters, or background replacements during video calls, and these effects are very popular with users. However, in these products, adding effects requires users to first access the effects menu and then select an effect, a rather cumbersome process. This is especially inconvenient during incoming calls, where users need to simultaneously answer the call and adjust the effects menu. Therefore, a faster way to add effects is urgently needed.

[0004] Current video call products typically display effect icons on the incoming call screen. Users click these icons to access an effects menu or beauty adjustment bar. To apply an effect, the user must first click the icon to open the effects editing menu, then select an effect thumbnail or adjust the beauty level within the menu. This process requires at least two clicks to activate the desired effect, making it cumbersome and time-consuming. Furthermore, the small clickable area of ​​the icons hinders quick operation. Therefore, it fails to meet the requirement of quickly activating effects during an incoming call.

[0005] [Application Content]

[0006] In view of this, embodiments of this application provide a method, apparatus, and terminal device for adding video effects, in order to solve the technical problem in the prior art that it is inconvenient and slow for users to add effects during video calls.

[0007] In a first aspect, embodiments of this application provide a method for adding video special effects, the method comprising the following steps: capturing a user's hand gestures and obtaining sliding information of the hand gestures; and applying special effects materials to the screen based on the sliding information of the hand gestures.

[0008] The solution provided in this embodiment differs from the existing technology, which requires at least two clicks for users to activate special effects during video calls, resulting in a cumbersome interaction process. The solution provided in this embodiment captures and processes the user's swipe gesture information, achieving the effect of adding special effects wherever the user's finger swipes.

[0009] In a preferred embodiment, the step of capturing the user's gesture and obtaining the sliding information of the gesture includes: capturing and recording in real time the starting coordinates of the user's finger in the two-dimensional xy coordinate system of the screen and the ending coordinates of the finger's sliding point; determining the sliding direction and sliding speed of the finger based on the starting coordinates and the ending coordinates of the finger; and determining the sliding gesture used by the user based on the sliding direction of the finger; wherein the sliding information includes the starting coordinates of the sliding point, the ending coordinates of the sliding point, the sliding gesture, the sliding direction, and the sliding speed.

[0010] The solution provided in this embodiment leaves swiping information when a user swipes their finger across the screen. The starting coordinates of the swipe start point and the ending coordinates of the swipe end point provide the basis for judging the swipe gesture, swipe direction, and swipe speed. The swipe gesture provides the way the user's finger swipes on the screen, the swipe direction provides the path the user's finger swipes on the screen, and the swipe speed provides the speed at which the user's finger swipes on the screen. By combining these swipe information, the area on the screen that needs to be replaced with special effects material can be accurately determined.

[0011] In a preferred embodiment, when the swipe gesture is a side-swipe gesture, the following steps are performed: Based on the starting point coordinate information and the ending point coordinate information of the finger, determine the coordinate displacement D of the finger on the screen and the preset minimum distance D. min The relationship between the magnitudes, and the sliding speed V and the preset minimum speed V. min The size relationship; when D > D min And V > V min At that time, the aforementioned special effects material will be activated.

[0012] The solution provided in this embodiment pre-sets a minimum distance D. min and minimum speed V min As a standard for judging whether a user's finger is effectively sliding, the system calculates the finger's coordinate displacement D and sliding speed V on the screen to determine whether the user's finger is actually performing a gesture operation, thus avoiding the problem of incorrectly adding special effects materials due to accidental touches by the user on the screen.

[0013] In a preferred embodiment, when the swipe gesture is an angled swipe gesture, the following steps are performed: a preset starting area is defined; wherein the maximum value of the horizontal coordinate of the starting area is X. max The maximum value of the x-axis is Y. max Based on the starting point coordinates of the finger, determine whether the starting point of the finger's slide falls within the starting point area; if the x-coordinate of the starting point of the finger's slide is X... s <Xmax And the ordinate Y s <Y max If the finger's sliding origin is within a preset origin area, then the coordinate displacement D of the finger on the screen is determined to be within a preset minimum distance D based on the origin coordinate information and the end coordinate information of the finger. min The relationship between the magnitudes, and the sliding speed V and the preset minimum speed V. min The size relationship; when D > D min V > V min Furthermore, the slope k of the line connecting the sliding start point and the sliding end point satisfies either 0.268 < |k| < 3.73 or 0.268 < |k|. -1 When |<3.73, the aforementioned special effects material is activated.

[0014] The solution provided in this embodiment pre-defines a starting area and a sliding area. The starting area serves as the criterion for determining whether a user's finger can trigger a sliding gesture recognition, while the sliding area serves as the criterion for determining whether a user's finger can trigger the application of special effects. By setting these two criteria, the system accurately determines whether a user's finger is performing a sliding gesture in the corner of the screen and the size of the area the finger is sliding. The sliding areas set at the four corners of the screen have an edge angle between 15° and 75°. When a user's finger performs a corner sliding gesture, it will cover part or all of the sliding area, thus satisfying the judgment requirements for corner sliding gestures and enabling the adaptive application of special effects.

[0015] In a preferred embodiment, the step of applying special effects material to the screen based on the sliding information of the gesture includes: dividing the screen into a first region and a second region based on the sliding information of the gesture; performing portrait segmentation on the portrait displayed on the screen; and replacing the background portion of the screen located in the first region with special effects material; wherein, the first region is the area of ​​the screen swept by the dividing line that is perpendicular to the line connecting the sliding start point and the sliding end point of the gesture and passes through the sliding end point, and the second region is the area of ​​the screen other than the first region; the sliding information includes the starting coordinate information of the sliding start point, the ending coordinate information of the sliding end point, the sliding gesture, the sliding direction, and the sliding speed.

[0016] The solution provided in this embodiment first performs portrait segmentation and screen segmentation to extract the portrait. The screen is then divided into a first area swiped by the finger and a second area not swiped by the finger. Only the background of the first area is replaced with special effects material. This method can capture the finger's position in real time during the swipe, thereby determining the first area where the background needs to be replaced. When replacing the background, the area occupied by the portrait still displays the portrait and is not replaced by the background, creating the effect of the foreground portrait remaining unchanged while the background changes. This achieves the functions of real-time capture of the user's swipe gestures, real-time calculation of the area swiped by the user's finger, and real-time replacement of the background of the area swiped by the user's finger.

[0017] In a preferred embodiment, the step of dividing the screen into a first region and a second region based on the sliding information of the gesture includes: creating the dividing line based on the starting coordinate information of the starting point of the sliding motion and the ending coordinate information of the ending point of the sliding motion; and dividing the area of ​​the screen swept by the dividing line along the sliding direction into the first region based on the dividing line and the sliding direction.

[0018] The solution provided in this embodiment utilizes the starting coordinates of the sliding start point and the ending coordinates of the sliding end point to determine the function of the dividing line used to segment the first and second regions in the xy coordinate system through simple calculations, thereby enabling the determination of the first region swept by the finger on the screen. This method consumes fewer computational resources, has a faster processing speed, and can adapt to the addition of special effects materials in scenarios with fast-moving gestures.

[0019] In a preferred embodiment, the step of performing portrait segmentation on the portrait displayed on the screen and replacing the background portion of the screen located in the first region with special effects material includes: downsampling the original image of the portrait using a bilinear interpolation algorithm to obtain a downsampled image; calculating the contour of the person in the downsampled image and upsampling the contour of the person using a bilinear interpolation algorithm to obtain the contour of the original image; segmenting the portrait from the original image based on the contour of the original image to obtain a first layer; replacing the background portion located in the first region with special effects material to obtain a second image; and overlaying the first image and the second image.

[0020] The solution provided in this embodiment downsamples the original image before performing portrait segmentation, thereby significantly reducing the computational and power consumption overhead of portrait segmentation. This allows deep learning portrait segmentation model algorithms that require a large amount of computation to run on mobile terminals with limited computing power and power consumption, meeting the frame rate requirements for video playback.

[0021] In a preferred embodiment, the level of addition of the special effects material is selected based on the area of ​​the first region or the screen ratio of the first region to the screen.

[0022] With the solution provided in this embodiment, users can choose the amount of special effects materials to add during the process of adding special effects materials. In terms of operation, this means that the level of special effects materials can be selected according to the size of the area on the screen that the user's finger slides. This operation method can meet users' needs for adding more and more complex special effects.

[0023] In a preferred embodiment, for every 10% increase in the screen occupancy of the first area, the level of the special effects material is increased by one level.

[0024] The solution provided in this embodiment can offer users a smoother experience in adding video effects.

[0025] In a preferred embodiment, when the screen occupancy of the first area is greater than 50% and the endpoint coordinates of the swipe endpoint of the gesture no longer change, the entire background of the screen is replaced with special effects material.

[0026] The solution provided in this embodiment is suitable for users who often want to completely replace the background after swiping their finger halfway through the process. Therefore, the method of this preferred embodiment can facilitate user operation and enhance user experience.

[0027] In a preferred embodiment, before the step of dividing the screen into a first region and a second region based on the sliding information of the gesture, the method includes: determining whether the sliding of the gesture is effective based on the sliding speed; if the sliding speed is greater than a first speed, replacing the background of the screen with overall special effects material; if the sliding speed is less than a second speed, not performing the step of applying special effects material to the screen based on the sliding information of the gesture; wherein the first speed is greater than the second speed.

[0028] The solution provided in this embodiment pre-sets two judgment conditions for judging the sliding speed. When the sliding speed is greater than the first speed, it can be assumed that the user needs to replace the entire background. When the sliding speed is less than the second speed, it can be assumed that the user has not performed a sliding gesture, and therefore there is no need to start the application effects material step.

[0029] In a preferred embodiment, after the steps of performing portrait segmentation on the portrait displayed on the screen and replacing the background portion of the screen located in the first area with special effects material, the method includes: determining the sliding direction of the gesture; if the sliding direction is positive away from the sliding starting point of the finger on the screen, adding new special effects material; if the sliding direction is negative towards the sliding starting point of the finger on the screen, restoring the previous special effects material.

[0030] The solution provided in this embodiment, by recognizing the sliding direction, offers users a backtracking operation. This allows users to freely choose their preferred special effects from multiple options without having to scroll past their favorite effects.

[0031] Concerned about enhancing the user experience.

[0032] In a preferred embodiment, the step of applying special effects materials to the screen based on the swiping information of the gesture includes: according to the swiping information of the gesture, pulling out a control panel at the starting point of the gesture on the screen in the direction of the user's swipe, wherein multiple special effects materials are displayed on the control panel; and applying the special effects materials to the screen.

[0033] The solution provided in this embodiment offers users a selective control panel, resulting in a better visual experience when replacing special effects materials and enabling them to select special effects materials that better meet their needs.

[0034] In a preferred embodiment, prior to the step of applying special effects material to the screen based on the swiping information of the gesture, the method includes: detecting the usage frequency of each special effects material, and sorting the presentation order of the special effects materials from most to least frequent.

[0035] The solution provided in this embodiment can customize the list order of commonly used special effects materials according to user habits, so that users can add special effects materials to the screen background more quickly each time they use the device, making it more convenient and providing a better user experience.

[0036] In a preferred embodiment, the number of special effects materials is less than 10.

[0037] The solution provided in this embodiment sets an upper limit on the number of materials in the backup library of special effects materials. These special effects materials appear in a loop when swiping, avoiding the problem that it is difficult for the user to return to the original state after swiping due to too many special effects materials.

[0038] Secondly, this application provides a video special effects adding device, the device comprising: an information acquisition module for capturing a user's gesture and acquiring sliding information of the gesture; and a special effects application module for applying special effects materials to the screen based on the sliding information of the gesture.

[0039] The solution provided in this embodiment differs from the cumbersome interaction process in the prior art, which requires at least two clicks for users to activate special effects during video calls. The solution provided in this embodiment captures and processes the sliding information of the user's gestures using an information acquisition module and a special effects application module, achieving the effect of adding special effects wherever the user's finger slides.

[0040] In a preferred embodiment, the information acquisition module includes: a recording unit, used to capture and record in real time the starting coordinate information of the user's finger sliding in the two-dimensional xy coordinate system of the screen and the ending coordinate information of the sliding end point; a calculation unit, used to determine the sliding direction and sliding speed of the finger based on the starting coordinate information and the ending coordinate information of the finger; and a judgment unit, used to determine the sliding gesture adopted by the user based on the sliding direction of the finger.

[0041] The solution provided in this embodiment records the sliding information generated when the user's finger slides on the screen. The calculation unit determines the area on the screen to add special effects materials by using the coordinates of the starting point and ending point of the user's finger when sliding. The judgment unit judges the sliding trajectory of the user's finger when the user uses different sliding gestures, so as to meet the need to add special effects materials in real time.

[0042] In a preferred embodiment, the special effects application module includes: a segmentation unit, used to segment the screen into a first region and a second region based on the swiping information of the gesture; and an application unit, used to perform portrait segmentation on the portrait displayed on the screen and replace the background portion of the screen located in the first region with special effects material.

[0043] The solution provided in this embodiment first calculates the size of the area swiped by the user's finger, then segments the screen into a first area swiped by the finger and a second area not swiped by the finger. The application unit first performs portrait segmentation, and then replaces the background of the first area with special effects material. This achieves the functions of capturing the user's swiping gesture in real time, calculating the area swiped by the user's finger in real time, and replacing the background of the area swiped by the user's finger in real time.

[0044] Thirdly, embodiments of this application provide a terminal device, including the video effects adding device as described in the second aspect.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium including a program or instructions that, when run on a computer, execute the method described in the first aspect.

[0046] Compared with the prior art, this technical solution has at least the following beneficial effects:

[0047] The video effects addition method, apparatus, and terminal device disclosed in this application replace the original button-clicking method with gestures to activate effects, reducing the complexity of user operation and allowing for quick activation of effects during video calls. It can perform regional portrait segmentation and background replacement or blurring based on the area swiped by the user's finger or the screen ratio, enhancing the product's fun, playability, and interactivity. Furthermore, it can downsample the original image before portrait segmentation, significantly reducing the computational and power consumption overhead of portrait segmentation. This allows computationally intensive deep learning portrait segmentation models to run on mobile terminals with limited computing power and power consumption, meeting the frame rate requirements for video playback. [Attached Image Description]

[0048] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the terminal device provided in Embodiment 1 of this application;

[0050] Figure 2 This is a schematic diagram of the steps of the video effects addition method provided in Embodiment 2 of this application;

[0051] Figure 3 This is a schematic diagram of step 100 in the video effects addition method provided in Embodiment 2 of this application;

[0052] Figure 4a This is the Cartesian coordinate system used in Step 100 of the video effects addition method provided in Embodiment 2 of this application when using a swipe gesture;

[0053] Figure 4b This application's embodiment 2 provides a schematic diagram of the operation principle when using a swipe gesture in step 100 of the video effects addition method;

[0054] Figure 5aThis is a schematic diagram of the starting area when using a angular swipe gesture in Step 100 of the video effects addition method provided in Embodiment 2 of this application;

[0055] Figure 5b This is a schematic diagram illustrating the operational principle of using a corner swipe gesture in Step 100 of the video effects addition method provided in Embodiment 2 of this application;

[0056] Figure 5c This is a diagram showing the effective sliding range when using a corner sliding gesture in Step 100 of the video effects addition method provided in Embodiment 2 of this application;

[0057] Figure 6 This is a schematic diagram of step 200 in the video effects addition method provided in Embodiment 2 of this application;

[0058] Figure 7 This is a schematic diagram of step 210 in the video effects addition method provided in Embodiment 2 of this application;

[0059] Figure 8 This is a schematic diagram of step 220 in the video effects addition method provided in Embodiment 2 of this application;

[0060] Figure 9 This is a schematic diagram of the contour image after portrait segmentation and processing in step 220 of the video effects addition method provided in Embodiment 2 of this application;

[0061] Figure 10 This is a schematic diagram of the steps before step 210 in the video effects addition method provided in Embodiment 2 of this application;

[0062] Figure 11 This is a schematic diagram illustrating the operation of switching special effects materials when the user's finger slides multiple times in the video special effects addition method provided in Embodiment 2 of this application;

[0063] Figure 12 This is a schematic diagram of another implementation of step 200 in the video effects addition method provided in Embodiment 2 of this application;

[0064] Figure 13a and Figure 13b This is an operational schematic diagram of another implementation of step 200 in the video effects addition method provided in Embodiment 2 of this application;

[0065] Figure 14 This is a schematic diagram of the steps before step 200 in the video effects addition method provided in Embodiment 2 of this application;

[0066] Figure 15This is a schematic diagram of the video effects adding device provided in Embodiment 3 of this application;

[0067] Figure 16 This is a schematic diagram of the information acquisition module in the video effects adding device provided in Embodiment 3 of this application;

[0068] Figure 17 This is a schematic diagram of the special effects application module in the video special effects addition device provided in Embodiment 3 of this application.

[0069] Figure label:

[0070] 1-Antenna;

[0071] 2-antenna;

[0072] 100 - Terminal equipment; 110 - Processor; 120 - External memory interface; 121 - Internal memory; 130 - Universal Serial Bus interface; 140 - Charging management module; 141 - Power management module; 142 - Battery; 150 - Mobile communication module; 160 - Wireless communication module; 170 - Audio module; 170A - Speaker; 170B - Receiver; 170C - Microphone; 170D - Headphone jack; 180 - Sensor module; 180A - Pressure sensor; 180 B - Gyroscope sensor; 180C - Barometric pressure sensor; 180D - Magnetic sensor; 180E - Accelerometer sensor; 180F - Proximity sensor; 180G - Near light sensor; 180H - Fingerprint sensor; 180J - Temperature sensor; 180K - Touch sensor; 180L - Ambient light sensor; 180M - Bone conduction sensor; 190 - Button; 191 - Motor; 192 - Indicator; 193 - Camera; 194 - Display screen; 195 - User identification module card interface;

[0073] 10-Information Acquisition Module; 11-Recording Unit; 12-Calculation Unit; 13-Judgment Unit;

[0074] 20 - Special effect application module; 21 - Segmentation unit; 22 - Application unit.

Detailed Implementation Methods

[0075] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0076] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0077] The following describes embodiments of terminal devices and methods for implementing terminal devices. The terminal device can be a mobile phone (also known as a smart terminal device), a tablet personal computer, a personal digital assistant, an e-book reader, or a virtual reality interactive device, etc. The terminal device can access various types of communication systems, such as Long Term Evolution (LTE) systems, future 5th Generation (5G) systems, new radio access technology (NR), and future communication systems such as 6G systems; it can also be a wireless local area network (WLAN), etc.

[0078] For ease of explanation, the following embodiments will use a smart terminal device as an example.

[0079] Example 1

[0080] like Figure 1 The diagram shown is a schematic diagram of a terminal device disclosed in Embodiment 1 of this application. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0081] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0082] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0083] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0084] The processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0085] In one embodiment, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0086] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In one embodiment, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0087] The I2S interface can be used for audio communication. In one embodiment, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In one embodiment, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0088] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In one embodiment, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In another embodiment, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0089] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In one embodiment, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In one embodiment, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0090] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In one embodiment, the processor 110 and the camera 193 communicate via the CSI interface to enable the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to enable the display function of the terminal device 100.

[0091] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In one embodiment, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0092] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge terminal device 100, and can also be used for data transfer between terminal device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other terminal devices, such as AR devices.

[0093] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0094] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In a wired charging embodiment, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In a wireless charging embodiment, the charging management module 140 receives wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also supply power to the terminal device via the power management module 141.

[0095] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In one embodiment, the power management module 141 can also be located within the processor 110. In another embodiment, the power management module 141 and the charging management module 140 can be housed in the same device.

[0096] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0097] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0098] The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In one embodiment, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In another embodiment, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0099] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In one embodiment, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and housed within the same device as the mobile communication module 150 or other functional modules.

[0100] The wireless communication module 160 can provide solutions for wireless communication applications on the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0101] In one embodiment, antenna 1 of terminal device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling terminal device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0102] Terminal device 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0103] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel, which may specifically include a foldable screen, a non-circular screen, etc. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In one embodiment, the terminal device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0104] Terminal device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0105] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In one embodiment, the ISP can be integrated into the camera 193.

[0106] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In one embodiment, the terminal device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0107] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when terminal device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0108] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. Thus, terminal device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0109] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0110] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0111] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of terminal device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of terminal device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0112] Terminal device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0113] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In one embodiment, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0114] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or make hands-free calls through the speaker 170A.

[0115] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 100 answers a phone call or voice message, the receiver 170B can be brought close to the listener's ear to receive the voice message.

[0116] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal device 100 may be equipped with at least one microphone 170C. In some embodiments, terminal device 100 may be equipped with two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal device 100 may be equipped with three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0117] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0118] Pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In one embodiment, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Terminal device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, terminal device 100 detects the intensity of the touch operation based on pressure sensor 180A. Terminal device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In one embodiment, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0119] The gyroscope sensor 180B can be used to determine the motion attitude of the terminal device 100. In one embodiment, the gyroscope sensor 180B can determine the angular velocity of the terminal device 100 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the terminal device 100 through reverse movement, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0120] The barometric pressure sensor 180C is used to measure air pressure. In one embodiment, the terminal device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0121] The magnetic sensor 180D includes a Hall effect sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In one embodiment, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0122] The 180E accelerometer can detect the magnitude of acceleration of the terminal device 100 in various directions (typically three axes). When the terminal device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the attitude of the terminal device, and can be applied to applications such as landscape / portrait switching and pedometers.

[0123] A distance sensor 180F is used to measure distance. The terminal device 100 can measure distance via infrared or laser. In one embodiment, during scene capture, the terminal device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0124] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The terminal device 100 emits infrared light outward through the LED. The terminal device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 may use the proximity sensor 180G to detect when a user holds the terminal device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and screen locking.

[0125] The ambient light sensor 180L is used to sense the ambient light intensity. The terminal device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light intensity. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touches.

[0126] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0127] Temperature sensor 180J is used to detect temperature. In one embodiment, terminal device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, terminal device 100 reduces the performance of the processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, terminal device 100 heats battery 142 to prevent abnormal shutdown of terminal device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, terminal device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0128] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of terminal device 100, in a different position than display screen 194.

[0129] In one embodiment, the touch screen composed of touch sensor 180K and display screen 194 can be located in the side area or folded area of ​​terminal device 100, and is used to determine the location and gesture of the user's touch when the user's hand touches the touch screen; for example, when the user holds the terminal device, he can click any position on the touch screen with his thumb, then touch sensor 180K can detect the user's click operation and transmit the click operation to the processor, and the processor determines that the click operation is used to wake up the screen.

[0130] The bone conduction sensor 180M can acquire vibration signals. In one embodiment, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In one embodiment, the bone conduction sensor 180M can also be integrated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals based on the vibration signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information based on the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0131] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Terminal device 100 can receive button input and generate key signal inputs related to user settings and function control of terminal device 100.

[0132] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0133] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0134] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the terminal device 100. The terminal device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In one embodiment, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0135] When a terminal device uses a notched or foldable screen, its touch display can include multiple touch display areas. For example, a foldable screen in its folded state includes a folded area, which can also respond to touch. However, existing technologies have significant limitations in the operation of specific touch display areas, lacking dedicated operations for specific touch display areas. Therefore, this application provides a gesture interaction method. In this method, the side or folded area of ​​the terminal device has a touch response area. The terminal device can acquire input events from this touch response area and, in response to the input events, trigger the execution of the corresponding operation commands to achieve gesture operations on the side or folded area of ​​the terminal device, thereby improving the user experience.

[0136] In the terminal device disclosed in Embodiment 1 of this application, the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal device executes the method described in Embodiment 2 of this application.

[0137] Example 2

[0138] like Figure 2 The illustration shows a method for adding video effects disclosed in Embodiment 2 of this application, which can be applied to incoming call status. The method includes the following steps:

[0139] Step 100: Capture the user's hand gestures and obtain the swiping information of the gestures;

[0140] Step 200: Apply special effects to the screen based on the swiping information of the gesture.

[0141] The video effects addition method in this embodiment differs from the cumbersome interaction process caused by the requirement of at least two clicks for users to activate effects during video calls in the prior art. The solution provided in this embodiment captures and processes the sliding information of the user's gestures, so that the effects are added wherever the user's finger slides.

[0142] like Figure 3 As shown, in the video effects addition method of this embodiment, step 100: capturing the user's hand gestures and obtaining the swiping information of the hand gestures includes:

[0143] Step 110: Capture and record in real time the starting coordinates of the user's finger as it slides in the two-dimensional xy coordinate system on the screen, as well as the ending coordinates of the finger's endpoint.

[0144] Step 120: Determine the sliding direction and speed of the finger based on the starting and ending coordinates of the finger.

[0145] Step 130: Determine the swipe gesture used by the user based on the direction of the finger swipe.

[0146] The sliding information includes the starting coordinates of the sliding start point, the ending coordinates of the sliding end point, the sliding gesture, the sliding direction, and the sliding speed.

[0147] In this embodiment of the video effects addition method, when a user slides their finger across the screen, it leaves sliding information. The starting point coordinates and ending point coordinates of the sliding start point provide the basis for judging the sliding gesture, sliding direction, and sliding speed. The sliding gesture provides the way the user's finger slides on the screen, the sliding direction provides the path the user's finger slides on the screen, and the sliding speed provides the speed at which the user's finger slides on the screen. By combining these sliding information, the area on the screen that needs to be replaced with special effects material can be accurately determined.

[0148] In the video effects addition method of this embodiment, the user's swipe gestures are divided into side swipe gestures and corner swipe gestures. Side swipe gestures are swipes that start from any of the four sides of the phone and move horizontally or vertically toward the center of the screen; corner swipe gestures are mainly swipes that move from the corner toward the center of the screen.

[0149] When the swipe gesture is a side swipe gesture, perform the following steps:

[0150] Step 140: Based on the starting and ending coordinates of the finger, determine the distance D of the finger's coordinate displacement on the screen relative to the preset minimum distance D. minThe magnitude relationship, and the sliding speed V and the preset minimum speed V min Size relationship;

[0151] When D > D min And V > V min At that time, proceed to step 180.

[0152] Taking a swipe from the side of the screen towards the center as an example, determine the distance between the finger's coordinate displacement on the screen, |Xs-Xe|, and the preset minimum distance D. min The relationship between the magnitudes of the sliding speed Vx and the preset minimum speed Vmin;

[0153] When |Xs-Xe|>D min And V x >V min If the sliding is successful, it is considered a valid slide.

[0154] Its core idea is to capture and record in real time the trajectory and speed of the user's finger's movement along the two-dimensional coordinates (x, y) on the screen. Figures 4a to 4b (As shown).

[0155] The judgment condition is: if the x-coordinate of the sliding start point is X s The value is 0 (preferably, a 10-pixel tolerance space can be reserved, the specific value of which depends on actual needs), and the ordinate Y is 0. s The range of the center point of the screen (maximum Y-axis length) max +200 pixels); the x-coordinate of the sliding endpoint. e greater than X s The coordinate displacement |Xs-Xe| is greater than D. min (Preferably, the value can be determined based on the application scenario, user habits, screen sensitivity, etc., with a reference value of 100 pixels), and the moving speed V x Greater than V min (Reference value: 200 pixels / second).

[0156] The above determination process is a judgment Figures 4a to 4b The image shows a horizontal finger dragging motion; based on the above idea, it can be extended to know the area of ​​the horizontal dragging region, thus facilitating the execution of subsequent application effects materials.

[0157] It should be noted that on Android phones, this swipe gesture may be the system's default Home button (start button) function, so in this scenario, attention should be paid to system-level gesture monitoring and capture.

[0158] Similarly, if a finger slides down from the top of the screen, the coordinate displacement is |Ys-Ye|, and the sliding speed is V. yThe remaining steps can be completed by substituting the conditions and steps described above.

[0159] The video effects addition method in this embodiment pre-sets a minimum distance D. min and minimum speed V min As a standard for judging whether a user's finger is effectively sliding, the system calculates the finger's coordinate displacement D and sliding speed V on the screen to determine whether the user's finger is actually performing a gesture operation, thus avoiding the problem of incorrectly adding special effects materials due to accidental touches by the user on the screen.

[0160] Taking the movement from the top right corner of the screen towards the center as an example, when the swipe gesture is an edge swipe gesture, the following steps are performed:

[0161] Step 150: Preset a starting region; wherein, the maximum value of the x-coordinate of the starting region is X. max The maximum value of the x-axis is Y. max ;

[0162] Step 160: Based on the starting point coordinates of the finger, determine whether the starting point of the finger's slide falls within the starting point area;

[0163] If the x-coordinate of the starting point of the finger's slide is X s <X max And the ordinate Y s <Y max If the finger's sliding starting point is located within the preset starting point area, then step Step 170 is executed.

[0164] Step 170: Based on the starting point coordinates and ending point coordinates of the finger, determine the distance D of the finger's coordinate displacement on the screen relative to the preset minimum distance D. min The magnitude relationship, and the sliding speed V and the preset minimum speed V min Size relationship;

[0165] When D > D min V > V min If the slope k of the line connecting the sliding start point and the sliding end point satisfies 0.268<|k|<3.73 or 0.268<|k-1|<3.73, then proceed to step 180.

[0166] At this time, coordinate displacement

[0167] Step 180: Activate the special effects material.

[0168] like Figures 5a to 5c As shown, when a user's finger performs an angular swipe gesture, the trigger condition for recognizing the centripetal movement of the angle is that the starting point O must be within a certain range. Figure 5bWithin the starting area of ​​the rectangular shape, 50 is a reference value, and the size of the starting area can be adjusted appropriately according to actual needs or screen sensitivity.

[0169] Angular centripetal sliding motion recognition is based on Figure 4a The judgment rule obtained after appropriately optimizing the judgment conditions for the swipe gesture shown in Figure 4c is as follows: Figure 5c Taking the midpoint A as the endpoint, let the coordinates of A be (X(A), Y(A)), and the starting point be (X(O), Y(O)). The judgment rule is as follows:

[0170] ① Coordinate displacement exceeds D min (Reference value: 100 pixels)

[0171]

[0172] ② The coordinate movement speed exceeds V min (Reference value: 200 pixels / second);

[0173] ③ The slope k of the straight line AO ​​ranges from (0.268, 3.73), which corresponds to an angle of displacement between 15° and 75°.

[0174] The area between them is shown in the OAB triangle region in Figure 4c. (The range of the triangle region can be adjusted appropriately according to actual needs).

[0175] Slide region recognition (i.e., isosceles right triangle region recognition): Based on the endpoint of the finger swipe, the following processing is performed: Figure 5c Taking midpoint A as an example, let A be the endpoint of the user's swipe. Draw a straight line with a slope of -1 through point A. The triangular region enclosed by this line and the coordinate axes is the swipe area. Because the slope of this line is -1, all interior angles with the coordinate axes are 45°. Figure 5c As shown, an isosceles right triangle is used to prepare for background replacement of the triangular area in the image.

[0176] Figure 5c This example only illustrates the case of one corner of the terminal device; the judgment rules for the other three corners can be based on... Figure 5c Based on the examples given, the final judgment rule is 0.268 < |k| < 3.73 or 0.268 < |k|. -1 |<3.73.

[0177] The video effects addition method in this embodiment pre-sets a starting area and a sliding area. The starting area serves as a criterion for determining whether the user's finger can trigger the sliding gesture recognition, and the sliding area serves as a criterion for determining whether the user's finger can trigger the application of special effects materials. By setting two criterion conditions, it is possible to accurately determine whether the user's finger has a sliding gesture in the corner of the screen and the size of the area slid by the finger. The sliding areas set at the four corners of the screen have an edge angle in the range of 15° to 75°. When the user's finger performs a corner sliding gesture, it will cover part or all of the sliding area, which can meet the judgment requirements of corner sliding gesture, and thus can adaptively apply special effects materials.

[0178] like Figure 6 As shown, in the video effects addition method of this embodiment, step 200: applying effects material to the screen according to the swiping information of the gesture, includes:

[0179] Step 210: Based on the swipe information of the gesture, divide the screen into a first area and a second area;

[0180] Step 220: Perform portrait segmentation on the image displayed on the screen, and replace the background in the first area with special effects material.

[0181] The first region is the area of ​​the screen that is perpendicular to the line connecting the starting point and the ending point of the gesture and passes through the dividing line of the ending point. The second region is the area on the screen other than the first region. The swiping information includes the starting point coordinates of the swiping start point, the ending point coordinates of the swiping end point, the swiping gesture, the swiping direction, and the swiping speed.

[0182] The video effects addition method in this embodiment first performs portrait segmentation and screen segmentation. The portrait is extracted, and the screen is divided into a first area swiped by the finger and a second area not swiped. Only the background of the first area is replaced with effects material. This method can capture the finger's position in real time during the swipe, thereby determining the first area where the background needs to be replaced. When replacing the background, the area occupied by the portrait still displays the portrait and is not replaced by the background, creating the effect of the portrait remaining unchanged while the background changes. This achieves the functions of real-time capture of the user's swipe gesture, real-time calculation of the area swiped by the user's finger, and real-time replacement of the background of the area swiped by the user's finger.

[0183] like Figure 7 As shown, in the video effects addition method of this embodiment, step 210: dividing the screen into a first region and a second region according to the swiping information of the gesture, includes:

[0184] Step 211: Create a dividing line based on the starting coordinates of the swipe's starting point and the ending coordinates of the swipe's ending point;

[0185] Step 212: Based on the dividing line and the sliding direction, divide the area of ​​the screen swept by the dividing line along the sliding direction into the first region.

[0186] The video effects addition method in this embodiment first calculates the size of the area swiped by the user's finger, then divides the screen into a first area swiped by the finger and a second area not swiped by the finger, and only replaces the background of the first area with special effects material, thus realizing the functions of capturing the user's swiping gesture in real time, calculating the area swiped by the user's finger in real time, and replacing the background of the area swiped by the user's finger in real time.

[0187] like Figure 8 As shown, in the video effects addition method of this embodiment, step 220: performing portrait segmentation on the portrait displayed on the screen, and replacing the background portion of the screen located in the first area with special effects material, including:

[0188] Step 221: Use bilinear interpolation to downsample the original image of the portrait to obtain the downsampled image;

[0189] Step 222: Calculate the outline of the person in the downsampled image (person mask), and use bilinear interpolation algorithm to upsample the outline of the person to obtain the outline of the original image.

[0190] Step 223: Segment the human figure from the original image based on its contour to obtain the first layer;

[0191] Step 224: Replace the background portion located in the first area with special effects material to obtain the second layer;

[0192] Step 225: Overlay the first layer with the second layer.

[0193] Specifically, in the method disclosed in this embodiment, taking the Google Android standard camera API 2.0 engine as an example, other camera engines can be used to output continuous image frame data. The frame data format can be a common format such as YUV-NV21 or RGB. First, the original image is downsampled using a bilinear interpolation algorithm, for example, downsampling the original 1920x1080 resolution image to a 320x180 resolution image, thus obtaining a downsampled image. Then, portrait segmentation processing is performed on the downsampled image. The deep learning model used for portrait segmentation is not limited in this embodiment; common models such as CNN, FCN / FCN+ / UNet, etc., can be used. After portrait segmentation, data with the outline (mask) of the person is output, which is essentially a single image frame, such as... Figure 9 As shown, the resolution of this contour is currently 320x180. Processing the low-resolution contour significantly reduces the computational complexity and power consumption of portrait segmentation. The output contour is upsampled to the resolution of the original image (1920x1080 in this embodiment) using bilinear interpolation. Since the upsampling is performed on a black and white image, the image loss rate of the upsampling interpolation is minimized. Based on the contour data, the portrait is "extracted" from the original image to obtain the first layer. This first layer is then overlaid and rendered on the GPU with the second layer, which has had its background replaced with special effects material, to obtain the final image with the replaced background.

[0194] The video effects addition method in this embodiment uses bilinear interpolation to downsample the original image frame by frame, proportionally reducing the resolution and computational cost. Then, it performs portrait segmentation, outputting the portrait's outline, and finally uses bilinear interpolation to upsample the outline back to the original image's resolution. This significantly reduces the power consumption and processing latency of portrait segmentation on mobile terminal devices, meeting the 30FPS frame rate requirement for video calls. By downsampling the original image before portrait segmentation, this method significantly reduces the computational and power consumption overhead of portrait segmentation, allowing computationally intensive deep learning portrait segmentation models to run on computationally and power-constrained mobile terminals, thus meeting the frame rate requirements for video playback.

[0195] In the video effects addition method of this embodiment, the addition level of the effects material is selected according to the area of ​​the first region or the screen ratio of the first region.

[0196] In this embodiment of the video effects addition method, during the process of adding effects materials, the user can choose the amount of effects materials to add. In terms of operation, this means that the level of effects materials can be selected according to the size of the area on the screen where the user's finger slides. This operation method can meet the user's more and more complex needs for adding effects.

[0197] In the video effects addition method of this embodiment, the addition level of the effects material increases by one level for every 10% swipe of the gesture.

[0198] The video effects addition method in this embodiment can provide users with a smoother video effects addition experience.

[0199] In practical applications, the above text can be used. Figure 4a The swipe gesture described in Figure 4c triggers a beautification algorithm by capturing the horizontal / vertical movement of the screen, using the proportion of the swipe area to the entire screen as the beautification level. For example, with level 10 beautification: if the swipe starts from the left or top and the swipe area occupies 10%, level 1 beautification is activated; if the swipe reaches the right side, level 10 beautification is activated, and so on.

[0200] Since beauty effects are implemented at different levels, it is recommended to use the side swipe gesture capture trigger method. However, if there are business requirements or product design factors, the corner swipe gesture capture method can also be used to trigger the effect, as the implementation ideas of the two are similar.

[0201] In the video effects addition method of this embodiment, when the screen ratio of the first area is greater than 50% and the endpoint coordinate information of the swipe endpoint of the gesture no longer changes, the entire background of the screen is replaced with special effects material.

[0202] The video effects addition method in this embodiment is convenient for users and enhances the user experience, since users often need to completely replace the background when they swipe their finger halfway.

[0203] like Figure 10 As shown, in the video effects addition method of this embodiment, before step 210: dividing the screen into a first region and a second region according to the swiping information of the gesture, the method includes:

[0204] Step 201: Determine whether the swipe gesture is effective based on the swipe speed;

[0205] If the sliding speed is greater than the first speed, proceed to step 202: replace the background of the screen with overall special effects material;

[0206] If the sliding speed is less than the second speed, then step 200 is not executed; wherein the first speed is greater than the second speed.

[0207] The video effects addition method in this embodiment has two preset judgment conditions for judging the sliding speed. The method combines the user's sliding speed to judge whether the sliding is effective. If the sliding speed is very fast and the sliding speed is greater than the first speed, it can be assumed that the user needs to replace the entire background. Then, the complete replacement or blurring of the background is enabled. If the sliding speed is less than the second speed, it can be assumed that the user has not performed a sliding gesture. Therefore, the step of activating the application effects material is not required.

[0208] See Figure 6 In the video effects addition method of this embodiment, after performing portrait segmentation on the portrait displayed on the screen and replacing the background material of the part of the screen located in the first area with special effects material, step 220 includes:

[0209] Step 230: Determine the swiping direction of the gesture;

[0210] If the swipe direction is positive and away from the starting point of the swipe on the screen, proceed to step 240: Add new special effects material;

[0211] If the swipe direction is in the opposite direction to the starting point of the finger swipe on the screen, proceed to step 250: restore the previous effect material.

[0212] like Figure 11 As shown, when a user performs a gesture, they can switch between special effects by swiping multiple times. That is, based on the rules mentioned above, swiping the finger again will change to the next special effect.

[0213] The video effects addition method in this embodiment provides users with a backtracking operation by recognizing the swipe direction. This allows them to cancel the screen's background replacement / blurring; for example, swiping left activates background replacement / blurring; swiping left again changes the background material; and swiping right (the reverse operation) restores the previous background material / real background. This allows users to freely choose their preferred effects from multiple options without worrying about accidentally swiping past their favorite, thus enhancing the user experience.

[0214] like Figure 12 , Figure 13a and Figure 13b As shown, in the video effects addition method of this embodiment, step 200 applies the effects material to the screen based on the swiping information of the gesture, including:

[0215] Step 210': Based on the swiping information of the gesture, pull out the control panel at the starting point of the gesture on the screen in the direction of the user's swipe. The control panel contains multiple special effects materials.

[0216] Step 220': Apply special effects materials to the screen.

[0217] The video effects addition method in this embodiment aims to achieve regional background replacement / blurring. It is not limited to the above-mentioned edge swipe gesture and corner swipe gesture capture methods. The core of this method is to use swipe gestures to quickly activate effects. The area of ​​the swipe area of ​​the edge / corner swipe can be used as the range of background replacement / blurring, providing users with a selective control panel. When replacing effect materials, the user's visual experience is better, and they can select effect materials that better meet their needs.

[0218] like Figure 14 As shown, in the video effects addition method of this embodiment, before step 200: applying the effects material to the screen based on the swiping information of the gesture, the method further includes:

[0219] Step 200': Detect the usage frequency of each special effects material, and sort the presentation order of the special effects materials according to the usage frequency from most to least.

[0220] The video effects addition method in this embodiment can customize the list order of commonly used effects materials according to user habits. For example, it can prioritize displaying the last used effects materials or sort them from most to least used according to frequency of use. This allows users to add effects materials to the screen background more quickly each time they use the device, making it more convenient and providing a better user experience.

[0221] In the video effects addition method of this embodiment, the number of effect materials is less than 10.

[0222] The video effects addition method in this embodiment sets an upper limit on the number of materials in the backup library of effects materials. These effects materials appear in a loop when swiping, avoiding the problem that it is difficult for the user to return to the original state after swiping due to too many effects materials.

[0223] The video effects addition method disclosed in this embodiment replaces the original interactive design of activating effects through button clicks with a swipe gesture. Based on the area or screen ratio of the region where the user's finger swipes across the screen, regional portrait segmentation and background replacement / blurring are performed to achieve regional background replacement / blurring. This allows for regional background replacement during video calls, where the user swipes to perform regional background replacement, dividing the area according to the position of the user's finger on the screen and the angle of the swipe gesture, without transmitting the video stream of the replaced area, thus saving video stream data transmission. The area / screen ratio of the region swiped by the user's finger can be used as the basis for beauty-level adjustments. Alternatively, the complete user finger swipe action can be used for one-click switching of background replacement materials, beauty-related materials, 3D-Animoji, or sticker materials. Furthermore, this method also uses a method for capturing and analyzing angular centripetal swipe gestures.

[0224] Example 3

[0225] like Figure 15 The image shown is a video effects adding device provided in Embodiment 3 of this application. The device includes:

[0226] The information acquisition module 10 is used to capture the user's gestures and acquire the swiping information of the gestures;

[0227] Special effects module 20 is used to apply special effects materials to the area on the screen through which the gesture moves, based on the swiping information of the gesture.

[0228] The video effects adding device in this embodiment differs from the prior art in that users need to click at least twice to enable effects during video calls, which results in a cumbersome interaction process. The solution provided in this embodiment uses the information acquisition module 10 and the effects application module 20 to capture and process the sliding information of the user's gesture, so that the effects can be added wherever the user's finger slides.

[0229] like Figure 16 As shown, in the video effects adding device of this embodiment, the information acquisition module 10 includes:

[0230] The recording unit 11 is used to capture and record in real time the starting coordinate information of the user's finger sliding in the two-dimensional xy coordinate system of the screen and the ending coordinate information of the sliding end point.

[0231] The calculation unit 12 is used to determine the sliding direction and sliding speed of the finger based on the starting point coordinate information and the ending point coordinate information of the finger.

[0232] The judgment unit 13 is used to determine the swipe gesture used by the user based on the swipe direction of the finger.

[0233] In this embodiment of the video effects adding device, the recording unit 11 is used to record the sliding information generated when the user's finger slides on the screen, the calculation unit 12 determines the area on the screen to add special effects materials by using the coordinates of the starting point and ending point of the user's finger when sliding, and the judgment unit 13 judges the sliding trajectory of the user's finger when the user uses different sliding gestures, so as to meet the needs of adding special effects materials in real time.

[0234] like Figure 17 As shown, in the video effects adding device of this embodiment, the effects application module 2 includes:

[0235] The segmentation unit 21 is used to segment the screen into a first region and a second region based on the swiping information of the gesture action;

[0236] Application unit 22 is used to perform portrait segmentation on the portrait displayed on the screen and replace the background of the part of the screen located in the first area with special effects material.

[0237] In this embodiment of the video effects adding device, the segmentation unit 21 first calculates the size of the area swiped by the user's finger, performs screen segmentation, and divides the screen into a first area swiped by the finger and a second area not swiped by the finger. The application unit 22 first performs portrait segmentation, and then replaces the background of the first area with special effects material, thereby realizing the functions of capturing the user's swiping gesture in real time, calculating the area swiped by the user's finger in real time, and replacing the background of the area swiped by the user's finger in real time.

[0238] Example 4

[0239] Embodiment 4 of this application provides a terminal device, including the video effects adding device as described in Embodiment 2 of this application.

[0240] Example 5

[0241] Embodiment 5 of this application provides a computer-readable storage medium including a program or instructions that, when run on a computer, execute the method described in Embodiment 1 of this application.

[0242] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0243] The video effects addition method, apparatus, and terminal device disclosed in this application replace the original button-clicking method with gestures to activate effects, reducing the complexity of user operation and allowing for quick activation of effects during video calls. It can perform regional portrait segmentation and background replacement or blurring based on the area swiped by the user's finger or the screen ratio, enhancing the product's fun, playability, and interactivity. Furthermore, it can downsample the original image before portrait segmentation, significantly reducing the computational and power consumption overhead of portrait segmentation. This allows computationally intensive deep learning portrait segmentation models to run on mobile terminals with limited computing power and power consumption, meeting the frame rate requirements for video playback.

[0244] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0245] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for adding video special effects, characterized in that, The method includes the following steps: When a video call is received, a video call notification interface is displayed; Before the call is connected, capture the user's swiping gesture on the video call notification interface and obtain the swiping information of the gesture. As the user swipes on the video call notification interface, special effects are applied to the area on the screen where the user's finger swipes, based on the swipe information of the gesture. The step of applying special effects material to the area on the screen swiped by the user's finger based on the swiping information of the gesture includes: Based on the swiping information of the gesture, the screen is divided into a first region and a second region; The human figure displayed on the screen is segmented, and the background portion of the screen located in the first area is replaced with special effects material.

2. The video effects adding method according to claim 1, characterized in that, The step of capturing the user's swiping gesture on the video call notification interface and obtaining the swiping information of the gesture includes: The system can capture and record in real time the starting coordinates of the user's finger and the ending coordinates of the finger's sliding point in the two-dimensional xy coordinate system on the screen. Based on the starting point coordinate information and the ending point coordinate information of the finger, determine the sliding direction and sliding speed of the finger; The system determines the user's swipe gesture based on the direction of the finger swipe. The sliding information includes the starting coordinates of the sliding start point, the ending coordinates of the sliding end point, the sliding gesture, the sliding direction, and the sliding speed.

3. The video effects adding method according to claim 2, characterized in that, When the swipe gesture is a side swipe gesture, the following steps are performed: Based on the starting point coordinates and ending point coordinates of the finger, determine the coordinate displacement D of the finger on the screen and its relationship to the preset minimum distance D. min The relationship between the magnitudes, and the sliding speed V and the preset minimum speed V. min Size relationship; When D > D min And V > V min At that time, the step of applying special effects material to the area on the screen that the user's finger swipes based on the swiping information of the gesture is executed.

4. The video effects adding method according to claim 2, characterized in that, When the swipe gesture is a corner swipe gesture, the following steps are performed: A preset starting region is defined; wherein the maximum value of the x-coordinate of the starting region is X. max The maximum value of the x-axis is Y. max ; Based on the starting point coordinate information of the finger, determine whether the starting point of the finger's sliding motion falls within the starting point area; If the x-coordinate of the starting point of the finger's slide is X s <X max And the ordinate Y s <Y max If so, it is determined that the starting point of the finger's slide is located within the preset starting point area; Based on the starting point coordinates and ending point coordinates of the finger, determine the coordinate displacement D of the finger on the screen and its relationship to the preset minimum distance D. min The relationship between the magnitudes, and the sliding speed V and the preset minimum speed V. min Size relationship; When D > D min V > V min Furthermore, the slope k of the line connecting the sliding start point and the sliding end point satisfies either 0.268 < |k| < 3.73 or 0.268 < |k|. -1 When |<3.73, the step of applying special effects material to the area on the screen where the user's finger swipes based on the swiping information of the gesture is executed.

5. The video effects adding method according to claim 1, characterized in that, The first region is the area of ​​the screen that is perpendicular to the line connecting the starting point and the ending point of the gesture and passes through the dividing line of the ending point. The second region is the area of ​​the screen other than the first region. The sliding information includes the starting coordinates of the sliding start point, the ending coordinates of the sliding end point, the sliding gesture, the sliding direction, and the sliding speed.

6. The video effects adding method according to claim 5, characterized in that, The step of dividing the screen into a first region and a second region based on the swiping information of the gesture includes: The dividing line is created based on the starting coordinates of the starting point of the gesture and the ending coordinates of the ending point. Based on the dividing line and the sliding direction, the area of ​​the screen swept by the dividing line along the sliding direction is divided into the first region.

7. The video special effects adding method according to claim 5, characterized in that, The steps of performing portrait segmentation on the portrait displayed on the screen and replacing the background portion of the screen located in the first area with special effects material include: The original image of the portrait is downsampled using a bilinear interpolation algorithm to obtain a downsampled image; Calculate the outline of the person in the downsampled image, and then use a bilinear interpolation algorithm to upsample the outline of the person to obtain the outline of the original image. The portrait is segmented from the original image based on its contour to obtain the first layer; Replace the background portion located in the first area with special effects material to obtain the second layer; Overlay the first layer with the second layer.

8. The video effects adding method according to claim 5, characterized in that, The level of the special effects material is selected based on the area of ​​the first region or the percentage of the screen occupied by the first region.

9. The video effects adding method according to claim 8, characterized in that, For every 10% increase in the screen occupancy of the first area, the level of the special effects material is increased by one level.

10. The video special effects adding method according to claim 5, characterized in that, When the screen occupancy of the first area is greater than 50% and the endpoint coordinates of the swipe gesture no longer change, the entire background of the screen is replaced with special effects material.

11. The video effects adding method according to claim 5, characterized in that, Before the step of dividing the screen into a first region and a second region based on the swiping information of the gesture, the method includes: The effectiveness of the gesture is determined based on the sliding speed. If the sliding speed is greater than the first speed, then the background of the screen will be replaced with a complete set of special effects material; If the sliding speed is less than the second speed, the step of applying special effects material to the area on the screen where the user's finger has swiped, based on the sliding information of the gesture, will not be performed. Wherein, the first speed is greater than the second speed.

12. The video effects adding method according to claim 5, characterized in that, After the steps of performing portrait segmentation on the portrait displayed on the screen and replacing the background portion of the screen located in the first area with special effects material, the process includes: Determine the sliding direction of the gesture; If the sliding direction is positive and away from the starting point of the finger's sliding on the screen, add new special effects material; If the sliding direction is in the opposite direction to the starting point of the finger's sliding on the screen, restore the previous effect material.

13. The video effects adding method according to claim 1, characterized in that, The step of applying special effects material to the area on the screen swiped by the user's finger based on the swiping information of the gesture includes: Based on the swiping information of the gesture, a control panel is pulled out at the starting point of the gesture on the screen and displayed in the area swiped by the user's finger, with multiple special effects materials set on the control panel; In response to the user's selection of the multiple special effects materials on the control panel, the selected special effects material is applied to the screen.

14. The video effects adding method according to claim 1, characterized in that, Prior to the step of applying special effects material to the screen based on the swiping information of the gesture, the following steps are included: The frequency of use of each special effect material is detected, and the presentation order of the special effect materials is sorted according to the frequency of use from most to least.

15. The video effects adding method according to claim 1, characterized in that, The number of special effects materials is less than 10.

16. The video effects adding method according to claim 2, characterized in that, After the process of applying special effects to the area on the screen where the user's finger swipes based on the swipe information of the gesture while the user is swiping on the video call notification interface, the method further includes: Replace the background of the screen with a complete set of special effects materials; As the user swipes on the video call notification screen again, new special effects are applied to the area on the screen where the user's finger is swiping, based on the swipe information of the gesture.

17. The video effects adding method according to claim 1, characterized in that, The video call notification interface has an answer button and a reject button.

18. A video special effects adding device, characterized in that, The device includes: The information acquisition module is used to display a video call notification interface when a video call is received; before connecting the call, it captures the user's swiping gesture on the video call notification interface and obtains the swiping information of the gesture. The special effects module is used to apply special effects materials to the area on the screen where the user's finger is swiping, based on the swiping information of the gesture action, when the user swipes on the video call notification interface. The step of applying special effects material to the area on the screen swiped by the user's finger based on the swiping information of the gesture includes: Based on the swiping information of the gesture, the screen is divided into a first region and a second region; The human figure displayed on the screen is segmented, and the background portion of the screen located in the first area is replaced with special effects material.

19. The video effects adding device according to claim 18, characterized in that, The information acquisition module includes: The recording unit is used to capture and record in real time the starting coordinate information of the user's finger sliding in the two-dimensional xy coordinate system on the screen and the ending coordinate information of the sliding end point. The calculation unit is used to determine the sliding direction and sliding speed of the finger based on the starting point coordinate information and the ending point coordinate information of the finger; The judgment unit is used to determine the swipe gesture used by the user based on the direction of the finger swipe.

20. The video special effects adding device according to claim 18, characterized in that, The special effect module includes: A segmentation unit is used to segment the screen into a first region and a second region based on the swiping information of the gesture action; The application unit is used to perform portrait segmentation on the portrait displayed on the screen and to replace the background of the portion of the screen located in the first area with special effects material.

21. A terminal device, characterized in that, Includes the video effects adding device as described in any one of claims 18 to 20.

22. A computer-readable storage medium, characterized in that, Includes a program or instructions that, when run on a computer, execute the method as described in any one of claims 1 to 17.

Images