Image processing method, device, storage medium and computer equipment

Automatically determine and add additional elements through image semantic segmentation technology, solve the problem of low image processing efficiency in the prior art, realize automatic processing, and improve efficiency.

CN110163861BActive Publication Date: 2024-11-29TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN201810755907.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2018-07-11
Publication Date
2024-11-29
Estimated Expiration
2039-07-23

AI Technical Summary

Technical Problem

The existing image processing methods require a lot of manual operations, resulting in low image processing efficiency and long-term time.

Method used

The target area and reference area are automatically determined through image semantic segmentation technology, and the target area and reference area are determined based on the positional relationship between the target area and the reference area in the multi-frame image frame. The additional element is automatically added to the subsequent image when the action is detected. in frame.

Benefits of technology

This greatly improves image processing efficiency, reduces the tedious steps of manual operation, and simplifies the image processing flow.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110163861B_ABST
    Figure CN110163861B_ABST
Patent Text Reader

Abstract

The present invention relates to an image processing method, device, storage medium and computer equipment, the method comprising: acquiring a captured image frame; determining a target area and a reference area obtained by image semantic segmentation in the acquired image frame; when the positional relationship between the target area and the reference area in the image frame with an earlier acquisition time meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame with a later acquisition time meets the action determination end condition, it is determined that an action that triggers the addition of an additional element is detected; acquiring the additional element when the action is detected; and adding the additional element to the image frame acquired after the image frame with a later acquisition time. The solution provided by the present application improves the efficiency of image processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image processing method, device, storage medium and computer equipment. Background Art

[0002] With the development of computer technology, image processing technology has also been continuously improving. Users can process images through professional image processing software to make the processed images perform better. Users can also use image processing software to attach materials provided by the image processing software to the image, so that the processed image can convey more information.

[0003] However, the current image processing method requires the user to expand the material library of the image processing software, browse the material library, select the appropriate material from the material library, adjust the position of the material in the image, and then confirm the modification to complete the image processing. Therefore, the current image processing method requires a lot of manual operations, which is time-consuming and leads to low efficiency of the image processing process. Summary of the invention

[0004] Based on this, it is necessary to provide an image processing method, apparatus, storage medium and computer equipment to address the problem of relatively low efficiency of current image processing.

[0005] An image processing method, comprising:

[0006] Get the captured image frame;

[0007] Determine a target region and a reference region obtained by image semantic segmentation in the acquired image frame;

[0008] When the positional relationship between the target area and the reference area in the image frame acquired earlier meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame acquired later meets the action determination end condition, it is determined that the action of triggering the addition of the additional element is detected;

[0009] acquiring additional elements upon detecting said action;

[0010] The additional element is added to an image frame acquired after the image frame acquired at a later time.

[0011] An image processing device, comprising:

[0012] An acquisition module, used for acquiring the collected image frames;

[0013] A determination module, used to determine the target area and the reference area obtained by image semantic segmentation in the acquired image frame;

[0014] A determination module, configured to determine that an action of triggering the addition of an additional element is detected when the positional relationship between the target area and the reference area in the image frame acquired earlier meets the action determination start condition and the positional relationship between the target area and the reference area in the image frame acquired later meets the action determination end condition;

[0015] The adding module is used to obtain additional elements when the action is detected; and add the additional elements to the image frame acquired after the image frame acquired at a later time.

[0016] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is caused to perform the following steps:

[0017] Get the captured image frame;

[0018] Determine a target region and a reference region obtained by image semantic segmentation in the acquired image frame;

[0019] When the positional relationship between the target area and the reference area in the image frame acquired earlier meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame acquired later meets the action determination end condition, it is determined that the action of triggering the addition of the additional element is detected;

[0020] acquiring additional elements upon detecting said action;

[0021] The additional element is added to an image frame acquired after the image frame acquired at a later time.

[0022] A computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs the following steps:

[0023] Get the captured image frame;

[0024] Determine a target region and a reference region obtained by image semantic segmentation in the acquired image frame;

[0025] When the positional relationship between the target area and the reference area in the image frame acquired earlier meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame acquired later meets the action determination end condition, it is determined that the action of triggering the addition of the additional element is detected;

[0026] acquiring additional elements upon detecting said action;

[0027] The additional element is added to an image frame acquired after the image frame acquired at a later time.

[0028] After acquiring the captured image frames, the above-mentioned image processing method, apparatus, storage medium and computer equipment automatically determine the target area and reference area obtained by image semantic segmentation in the acquired image frames, and then determine whether there is an action that triggers the addition of additional elements based on the positional relationship between the target area and the reference area in multiple image frames. In this way, when the action is determined, the additional elements can be automatically added to the subsequently acquired image frames, avoiding the tedious steps of manual operation and greatly improving the efficiency of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0030] Figure 2 is a schematic flow chart of an image processing method in one embodiment;

[0031] Figure 3 It is a schematic diagram of the principle of processing the acquired image frame by the image semantic segmentation model in a specific embodiment;

[0032] Figure 4 is a schematic diagram of segmenting a hand region from an acquired image frame in one embodiment;

[0033] Figure 5 A schematic diagram of an image frame satisfying an action determination start condition in one embodiment;

[0034] Figure 6 A schematic diagram of an image frame that meets an action determination end condition in one embodiment;

[0035] Figure 7 A schematic diagram of an image frame with additional elements added in one embodiment;

[0036] Figure 8 is a flowchart of an image processing method in a specific embodiment;

[0037] Fig. 9 is a module structure diagram of an image processing device in one embodiment;

[0038] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] Figure 1FIG. 1 is an application environment diagram of an image processing method in an embodiment. Figure 1 , the image processing method is applied to an image processing system. The image processing system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected via a network. Both the terminal 110 and the server 120 can execute the image processing method. The terminal 110 can be a desktop terminal or a mobile terminal, and the mobile terminal can be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server 120 can be an independent server or a server cluster composed of multiple independent servers.

[0041] The terminal 110 can obtain the acquired image frame, which can be acquired by the terminal 110 through a built-in image acquisition device or an externally connected image acquisition device. The built-in image acquisition device can specifically be the front camera or the rear camera of the terminal 110; the image frame can also be acquired by other computer devices and sent to the terminal 110. The terminal 110 can then determine the target area and the reference area obtained by image semantic segmentation in the acquired image frame, and when the positional relationship between the target area and the reference area in the image frame with an earlier acquisition time meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame with a later acquisition time meets the action determination end condition, it is determined that the action of triggering the addition of the additional element is detected. In this way, the terminal 110 can obtain the additional element when the action is detected, and add the obtained additional element to the image frame acquired after the image frame with a later acquisition time.

[0042] The terminal 110 may also send the acquired image frames to the server 120. When the server 120 determines that the positional relationship between the target area and the reference area in the image frame with an earlier acquisition time satisfies the action determination start condition, and the positional relationship between the target area and the reference area in the image frame with a later acquisition time satisfies the action determination end condition, the terminal 110 is notified that an action triggering the addition of additional elements is detected. The terminal 110 then acquires the additional elements and adds the acquired additional elements to the image frames acquired after the image frame with a later acquisition time.

[0043] The terminal 110 may also send the acquired image frames to the server 120, and when the positional relationship between the target area and the reference area in the image frame with an earlier acquisition time meets the action determination start condition and the positional relationship between the target area and the reference area in the image frame with a later acquisition time meets the action determination end condition, the server 120 determines that an action triggering the addition of additional elements is detected, and adds the elements, adds the acquired additional elements to the image frames acquired after the image frame with a later acquisition time, and then feeds back the image frames after the additional elements are added to the terminal 110.

[0044] Figure 2FIG. 1 is a flow chart of an image processing method in an embodiment. This embodiment mainly uses the application of the image processing to a computer device as an example. The computer device may be Figure 1 The terminal 110 or the server 120 in FIG. Figure 2 , the method specifically comprises the following steps:

[0045] S202, acquiring the captured image frame.

[0046] The image frame is data obtained by imaging the imaging target through the physical imaging principle.

[0047] In one embodiment, when the computer device is a terminal, the terminal can specifically capture image frames at a fixed or dynamic frame rate to obtain the captured image frames. The fixed or dynamic frame rate can form a continuous dynamic picture when the image frames are played at the fixed or dynamic frame rate.

[0048] In one embodiment, when the computer device is a terminal, the terminal can capture image frames in the current field of view of the image capture device through a built-in or externally connected image capture device, and obtain the captured image frames. The field of view of the image capture device can change due to changes in the posture and position of the terminal. The image capture device of the terminal can specifically be a front camera or a rear camera.

[0049] In one embodiment, when the computer device is a terminal, the terminal can capture image frames through a shooting mode provided by a social application to obtain the captured image frames. The social application is an application that can perform network social interaction based on a social network. The social application includes an instant messaging application, an SNS (Social Network Service) application, a live broadcast application, or a photo application.

[0050] In one embodiment, when the computer device is a terminal, the terminal can receive an image frame sent by another terminal and captured by another terminal, and obtain the received image frame. For example, when the terminal establishes a video session through a social application running on the terminal, it receives the captured image frame sent by the terminal corresponding to the other session party.

[0051] In one embodiment, when the computer device is a terminal, the terminal can collect image frames through the shooting mode provided by the live broadcast application, and use the collected image frames as live broadcast data to broadcast live through the live broadcast application. The terminal can also receive image frames sent by another terminal and collected through the shooting mode provided by the live broadcast application, and use the received image frames as live broadcast data to play the live broadcast initiated by other users through the live broadcast application through the live broadcast application.

[0052] In one embodiment, when the computer device is a server, the terminal in the aforementioned embodiment can upload the acquired image frame to the server, so that the server acquires the acquired image frame.

[0053] In a specific embodiment, the computer device is a terminal. A video recording application is installed on the terminal. The terminal can run the video recording application according to a user instruction, call a camera built into the terminal through the video recording application to capture image frames, and when capturing image frames, obtain the captured image frames in real time according to the capture timing of the image frames.

[0054] In the above embodiment, the frame rate at which the computer device acquires image frames is less than or equal to the frame rate at which the image acquisition device acquires image frames.

[0055] S204, determining a target region and a reference region obtained by image semantic segmentation in the acquired image frame.

[0056] Among them, image semantic segmentation is to segment the pixels in the image according to the different semantics expressed. Image semantic segmentation is used to realize the division of the image into multiple pixel areas according to semantics. In essence, image semantic segmentation realizes the classification of the image at the pixel level, and the semantic annotation of the entire image is realized by classifying the pixels. It should be noted that the classification unit is not limited in the embodiment of the present invention, and it can be pixel-by-pixel classification or image block classification. An image block includes multiple pixels.

[0057] The target area is the area in the image frame that is used as the target detection action. The reference area is the area in the image frame that is used as the reference detection action. In different image frames, the target area is a dynamic area, and the reference area is a static area. The positional relationship between the target area and the reference area is different in different image frames. It can be understood that the static area here is not absolutely static, but is a static area relative to the target area.

[0058] For example, suppose that when the camera is capturing image frames, the user makes a head-lifting action. Since head-lifting is a continuous action, in a series of image frames captured by the camera when the user is using his hand to lift his head, the hand is the execution part of the head-lifting action, so the hand area is the target area, which changes dynamically in different image frames; the face is the reference part of the head-lifting action, so the face area is the reference area, which is static relative to the hand. In this scenario, the target area is a dynamic area, and the reference area is a relatively static area.

[0059] For another example, when the camera is capturing image frames, the user makes a jumping action. Since jumping is a continuous action, in a series of image frames captured by the camera when the user is jumping, the human body is the execution part of the jumping action, so the human body area is the target area, which changes dynamically in different image frames. For the convenience of calculation, the foot area (local area of ​​the human body) can also be selected as the target area. The ground is the reference part of the jumping action, so the ground area is the reference area. In this scenario, the reference area is an absolutely static area.

[0060] Specifically, the terminal may encode the acquired image frame into a semantic segmentation feature matrix, then decode the semantic segmentation feature matrix to obtain a semantic segmentation image, and then segment the target area from the semantic segmentation image according to the pixels belonging to the target category, and segment the reference area from the semantic segmentation image according to the pixels belonging to the reference category. The pixels in the semantic segmentation image have pixel values ​​representing the classification category to which they belong, and the pixels encoded therefrom correspond to the pixels in the image frame.

[0061] Those skilled in the art can understand that the semantic segmentation feature matrix is ​​a low-dimensional expression of the semantic features of the image content in the image frame, covering the semantic feature information of the entire image frame. The semantic segmentation image is an image segmented into several non-overlapping areas with certain semantics. The pixel values ​​of the pixels in the semantic segmentation image are used to reflect the classification category to which the corresponding pixels belong. The classification of pixels can be binary or multi-classified. Binary classification of pixels, such as pixels corresponding to roads and other pixels in a map image. Multi-classification of pixels, such as pixels corresponding to the sky, pixels corresponding to the earth, and pixels corresponding to people in a landscape map. The image size of the semantic segmentation image is consistent with the image size of the original image frame. In this way, it can be understood that the acquired image frame is classified pixel by pixel, and the category affiliation of each pixel in the acquired image frame can be obtained according to the pixel values ​​of the pixels in the semantic segmentation image.

[0062] S206, when the positional relationship between the target area and the reference area in the image frame with an earlier acquisition time meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame with a later acquisition time meets the action determination end condition, it is determined that the action triggering the addition of the additional element is detected.

[0063] Among them, the action determination start condition is a constraint condition for determining that there is a probability of starting to execute a specific action. The action determination end condition is a constraint condition for the probability of executing a specific action. Since action is a continuous process, it can be understood that the detected action can only be determined when both the action determination start condition and the action determination end condition are met.

[0064] For example, suppose that when the camera is capturing image frames, the user makes a head-lifting action. Since head-lifting is a continuous action, it can only be determined that there is a head-lifting action when it is detected that the user has started to lift his head and is still lifting his head. If the user stops the action immediately after starting to lift his head, it cannot be considered that the head-lifting action is detected.

[0065] For another example, suppose that when the camera is capturing image frames, the user makes a jumping action. Since jumping is a continuous action, the jumping action can only be determined when the user jumps and leaves the ground (jumping). If the user stops immediately after jumping and does not leave the ground, the jumping action cannot be detected.

[0066] The action of triggering the addition of additional elements is the action of triggering the addition of additional elements in the captured image frame. The actions that trigger the addition of additional elements include the head-lifting action, the face-covering action, or the chin-touching action. The additional elements are used to add additional data to the image frame. The additional elements can specifically be decorative elements, such as pendants. Decorative elements are data that can be displayed in a visual form for decoration. Additional elements include data displayed in an image frame to modify the image content. Additional elements include masks, armor, ribbons, blue sky or white clouds, etc. The types of additional elements are not limited in the embodiments of the present invention. Additional elements can be dynamic data, such as dynamic pictures; they can also be static data, such as static pictures.

[0067] In one embodiment, the action that triggers the addition of the additional element may be one or more. When there are multiple actions that trigger the addition of the additional element, different actions may correspond to the same action determination start condition, or correspond to the same action determination end condition. These multiple actions may trigger the addition of a unified additional element at a unified position in the image frame, or may trigger the addition of a unified additional element at different positions in the image frame, or may trigger the addition of different additional elements at different positions in the image frame.

[0068] It can be understood that the chronological relationship involved in the acquisition time earlier and the acquisition time later here refers to the acquisition time of the image frame whose positional relationship between the target area and the reference area meets the action determination start condition, and before the acquisition time of the image frame whose positional relationship between the target area and the reference area meets the action determination end condition.

[0069] S208, acquiring additional elements when an action is detected.

[0070] Specifically, when the terminal detects an action, it may query the previously established correspondence between the action and the additional element, query the additional element corresponding to the detected action according to the correspondence, and obtain the queried additional element.

[0071] In one embodiment, the number of additional elements corresponding to the action may be one or more. When the number of additional elements corresponding to the action may be more than one, the terminal may randomly select an additional element from the multiple additional elements, or may select an additional element matching the user tag of the currently logged-in user identifier according to the user tag.

[0072] S210, adding the additional element to the image frame acquired after the image frame acquired at a later time.

[0073] It can be understood that the image frame with a later acquisition time here is the image frame in which the positional relationship between the target area and the reference area meets the action determination end condition, and is the image frame acquired after the image frame in which the positional relationship between the target area and the reference area meets the action determination start condition.

[0074] Specifically, after determining that the action that triggers the addition of the additional element is detected, the computer device may use the image frame whose positional relationship between the target area and the reference area satisfies the action determination end condition (that is, the image frame whose acquisition time is later) as the demarcation frame, and add the additional element to the image frame whose acquisition time is later than the demarcation frame. The image frame to which the additional element is added may be all the image frames acquired after the acquisition time of the demarcation frame, or may be part of the image frames acquired after the acquisition time of the demarcation frame, and may also include the demarcation frame itself.

[0075] In a specific embodiment, the computer device is a terminal. A video recording application is installed on the terminal. The terminal can run the video recording application according to user instructions, call the camera built into the terminal through the video recording application to capture image frames, and when capturing image frames, acquire the captured image frames in real time according to the acquisition timing of the image frames. It can be understood that the acquisition of image frames by the camera is a real-time and continuous process, and the acquisition of the captured image frames by the terminal is also a real-time and continuous process. After acquiring each image frame, the terminal determines whether the positional relationship between the target area and the reference area in the image frame meets the action determination start condition; when the action determination start condition is met, the next image frame acquired is determined whether it meets the action determination end condition; when the action determination end condition is met, additional elements are added starting from the next image frame acquired (which may include the image frame that currently meets the action determination end condition).

[0076] For example, the terminal collects a series of image frames P1, P2...P i , P i+1 …P n The terminal determines that the positional relationship between the target area and the reference area in the image frame P2 meets the action determination start condition, and determines that the image frame P iThe positional relationship between the target area and the reference area satisfies the action determination end condition; and then it can be determined that the action of triggering the addition of additional elements is detected. Then, the terminal can i or P i+1 Start adding additional elements.

[0077] The above-mentioned image processing method, after acquiring the captured image frame, automatically determines the target area and reference area obtained by image semantic segmentation in the acquired image frame, and then determines whether there is an action that triggers the addition of additional elements based on the positional relationship between the target area and the reference area in multiple image frames. In this way, when the action is determined, the additional elements can be automatically added to the subsequently acquired image frames, avoiding the tedious steps of manual operation and greatly improving the efficiency of image processing.

[0078] In one embodiment, S204 includes: inputting the acquired image frame into an image semantic segmentation model; outputting a target area probability distribution matrix and a reference area probability distribution matrix through the image semantic segmentation model; determining the target area in the acquired image frame according to the target area probability distribution matrix; and determining the reference area in the acquired image frame according to the reference area probability distribution matrix.

[0079] Among them, the image semantic segmentation model is a machine learning model that has semantic segmentation function after training. The full name of machine learning is Machine Learning, or ML for short. Machine learning models can have specific capabilities through sample learning. Machine learning models can use neural network models, support vector machines, or logistic regression models. Neural network models include convolutional neural networks, etc.

[0080] In this embodiment, the image semantic segmentation model is specifically a neural network model. The neural network model can specifically be a convolutional neural network model (CNN). The convolution layer (Convolution Layer) of the convolutional neural network model includes multiple convolution kernels (Convolution Kernel). The convolution kernel is an operator that the convolution layer performs convolution operations on the input. After each convolution and convolution operation on the input, an output can be obtained. The pooling layer (Pooling Layer) of the neural network model is also called a sampling layer, which is used to compress the input, and usually has two forms of mean pooling (Mean Pooling) and maximum pooling (MaxPooling). Pooling can be regarded as a special convolution process.

[0081] The image semantic segmentation model can be generally understood as a classifier, which is used to classify the pixels included in the input image frame pixel by pixel. The number of classification categories of the image semantic segmentation model can be customized during training. In the present embodiment, the image semantic segmentation model is set as a multi-classifier, and the classification categories include three types: target category, reference category and background category. The pixels of the model input image frame that belong to the target category are the pixels of the target area, the pixels that belong to the reference category are the pixels of the reference area, and the pixels that belong to the background category are the pixels of the background area. In this way, the pixels can be divided according to the category to which the pixels belong, and the target area and reference area in the acquired image frame are determined.

[0082] For example, when the action that triggers the addition of additional elements is the head-lifting action, the target category is the hand category, and the reference category is the face category. The pixels belonging to the hand category in the acquired image frame are the pixels in the hand area, and the pixels belonging to the face category are the pixels in the face area. In this way, the pixels can be divided according to the categories to which they belong, and the hand area and the face area in the acquired image frame can be determined.

[0083] The matrix elements of the target region probability distribution matrix have probability values ​​representing the target category and correspond to the pixels in the image frame of the input model. In other words, assuming that the image frame of the input model is 2*2, the target region probability distribution matrix is ​​also 2*2, and the value of the matrix element at the matrix position (m,n) is the probability that the pixel at the pixel position (m,n) in the image frame belongs to the target category. Among them, the matrix (image frame) has the matrix position (pixel position) in the upper left corner as (0,0).

[0084] Similarly, the matrix elements of the reference region probability distribution matrix have probability values ​​representing the reference category and correspond to the pixels in the image frame of the input model. In other words, assuming that the image frame of the input model is 2*2, then the target region probability distribution matrix is ​​also 2*2, and the value of the matrix element at the matrix position (m,n) is the probability that the pixel at the pixel position (m,n) in the image frame belongs to the reference category.

[0085] Specifically, the terminal can input the acquired image frame into a pre-trained image semantic segmentation model, and output the target area probability distribution matrix and the reference area probability distribution matrix through the image semantic segmentation model. The terminal can then determine the area surrounded by the pixel points corresponding to the matrix elements with probability values ​​greater than the preset probability in the target area probability distribution matrix as the target area, and determine the area surrounded by the pixel points corresponding to the matrix elements with probability values ​​greater than the preset probability in the reference area probability distribution matrix as the reference area. The preset probability is a pre-set demarcation value for determining whether it is classified as the current category.

[0086] Figure 3FIG. 1 is a schematic diagram showing the principle of processing an acquired image frame by an image semantic segmentation model in a specific embodiment. Figure 3 , the image semantic segmentation model is a U-shaped symmetric model, and the output of the previous network layer is used as the input of the network layer at the corresponding position through a skip connection. The input of the image semantic segmentation model is the feature map of the acquired image frame (such as the RGB three-channel feature map). The network layer in the image semantic segmentation model operates the feature map of the input layer to obtain the feature map output. The output of the image semantic segmentation model can be a semantic segmentation image or a probability distribution matrix, which is determined by the samples and labels during training. Among them, in the figure, m*n*k (such as 3*256*256, or 64*256*256), m represents the number of feature maps, and n*k represents the size of the feature map. It can be understood that the parameters in the figure are examples and do not limit the actual model parameters used. The operations of the network layer on the feature map include: convolution change (Convolution), normalization change (BatchNorm), activation change (ReLU), maximum pooling (MaxPool) and upsampling (Upsampling), etc.

[0087] In the above embodiment, after the image frame is acquired, the image frame is automatically input into the trained machine learning model, and the target area and the reference area are determined according to the target area probability distribution matrix and the reference area probability distribution matrix output by the machine learning model. Among them, the matrix elements in the probability distribution matrix have probability values ​​indicating that the corresponding pixel points in the image frame belong to a specific classification category, so that the target area can be automatically determined according to the pixel points belonging to the target category, and the reference area can be determined according to the pixel points belonging to the reference category, which improves the accuracy of image area division and lays the foundation for the subsequent judgment of whether the action determination start condition or the action determination end condition is met.

[0088] In one embodiment, the target area is a hand area; the reference area is a facial area. The image processing method further includes: determining the gesture type corresponding to the hand area in the acquired image frame. When the gesture type is a gesture type that triggers the addition of additional elements, when the positional relationship between the target area and the reference area in the image frame before the acquisition time meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame after the acquisition time meets the action determination end condition, then it is determined that the action of triggering the addition of additional elements is detected, including: when the positional relationship between the hand area and the facial area in the image frame before the acquisition time meets the action determination start condition, and the positional relationship between the hand area and the facial area in the image frame after the acquisition time meets the action determination end condition, then it is determined that the action of triggering the addition of additional elements is detected.

[0089] Among them, the hand and the face are both limbs of a living being (human or animal). The hand area is the area where the hand is located. The hand area can be the area enclosed within the hand outline, or it can be a regular area that includes the hand and has a high hand ratio. The face area can be the area enclosed by the face outline, or it can be a regular area that includes the face and has a high face ratio. A gesture is a movement made by a user through his hand. The gesture type is the type of gesture in the acquired image frame.

[0090] Figure 4 FIG. 1 is a schematic diagram showing a method of segmenting a hand region from an acquired image frame in an embodiment. Figure 4 (a) is an acquired image frame, and the terminal can determine the hand region 401 in the image frame by image semantic segmentation. Figure 4 (b) is an image obtained by segmenting the hand area according to a regular shape from the acquired image frame including the hand area.

[0091] It can be understood that compared to directly identifying the gesture type corresponding to the hand area in the acquired original image, segmenting the hand area from the acquired image and then identifying the segmented hand area can avoid the problem of inaccurate recognition when the hand area occupies a small proportion of the entire image, and can reduce the interference of the background area relative to the hand area in the original image on the recognition of the gesture type of the gesture in the hand area, thereby improving the recognition accuracy.

[0092] Specifically, the computer device can use a pre-trained gesture recognition model to identify the gesture type to which the gesture belongs in the image frame. The hand area is intercepted from the acquired image frame to obtain a hand image and input into the gesture recognition model. The features corresponding to the hand image are calculated through the hidden layer in the gesture recognition model, and the gesture type of the gesture in the hand image is output. When the identified gesture type is a gesture type that triggers the addition of additional elements, it is determined whether the positional relationship between the hand area and the facial area in the acquired image frame meets the action determination start condition. Otherwise, it is determined whether the gesture type corresponding to the hand area in the next acquired image frame is a gesture type that triggers the addition of additional elements.

[0093] Furthermore, when the terminal recognizes that the gesture type corresponding to the hand area in a certain frame of image acquired is the gesture type that triggers adding additional elements, and the positional relationship between the hand area and the facial area in the image frame meets the action judgment start condition, the terminal continues to determine whether the gesture type corresponding to the hand area in the next frame of image acquired after the image frame is the gesture type that triggers adding additional elements, and when it determines that the gesture type corresponding to the hand area in the next frame of image is the gesture type that triggers adding additional elements, the terminal continues to determine whether the positional relationship between the hand area and the facial area in the next frame of image meets the action judgment end condition, until it detects that the gesture type corresponding to the hand area in another frame of image acquired later is the gesture type that triggers adding additional elements, and the positional relationship between the hand area and the facial area in the image frame meets the action judgment end condition, then it is determined that the action of triggering adding additional elements has been detected.

[0094] Among them, the gesture recognition model is a machine learning model. When the gesture type that triggers the addition of additional elements pre-set by the computer device is unique, the gesture recognition model is a binary classification model. The image samples used to train the binary classification model include positive samples of the gesture type that triggers the addition of additional elements, and negative samples of the gesture type that does not trigger the addition of additional elements. When the gesture types that trigger the addition of additional elements pre-set by the computer device are diverse, the gesture recognition model is a multi-classification model. The image samples used to train the multi-classification model include samples of each gesture type that triggers the addition of additional elements. The gesture recognition model can specifically use the ConvNet Configuration model as the initial model, train the initial model according to the training samples, and obtain model parameters suitable for gesture recognition.

[0095] In one embodiment, the computer device may also perform feature matching between a hand image obtained by intercepting a hand area from the acquired image frame and a hand image template belonging to a gesture type that triggers adding additional elements. When the match is successful, it is determined that the gesture type corresponding to the hand area in the acquired image frame is a gesture type that triggers adding additional elements.

[0096] In the above embodiment, in the scenario where the specific target area is the hand area and the reference area is the facial area, only when it is identified that the gesture type corresponding to the hand area in the acquired image frame is the gesture type that triggers the addition of additional elements, the action determination start condition or the action determination end condition is determined whether it is met, thereby avoiding the waste of resources caused by determining the action determination start condition or the action determination end condition under an invalid gesture and improving image processing efficiency.

[0097] In one embodiment, when the positional relationship between the target area and the reference area in the image frame with an earlier acquisition time meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame with a later acquisition time meets the action determination end condition, it is determined that the action of triggering the addition of additional elements is detected, including: starting timing from the time when the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition; when the timing duration does not reach a preset duration and the positional relationship between the target area and the reference area in the image frames that continue to be acquired meets the action determination end condition, it is determined that the action of triggering the addition of additional elements is detected; wherein the acquired image frame is the image frame with an earlier acquisition time, and the image frame that continues to be acquired is the image frame with a later acquisition time.

[0098] It can be understood that an action is not only continuous but also coherent. In layman's terms, it is completed continuously after the action starts. For example, for the head-twisting action, only when it is detected that the user starts to lift his head and continues to lift his head continuously (that is, it is detected that the head is being lifted within a certain time range), can it be determined that there is a head-twisting action; and if the user stops the action immediately after starting to lift his head, and waits for a long time before continuing to lift his head, then it cannot be considered that the head-twisting action is detected. For another example, for the jumping action, only when it is detected that the user jumps and the user leaves the ground continuously after jumping (jumping), can it be determined that there is a jumping action; and if the user stops the action immediately after jumping and does not leave the ground, and waits for a long time before leaving the ground, then it cannot be considered that the jumping action is detected.

[0099] Specifically, when the computer device acquires each image frame, it determines whether the image frame meets the action determination start condition, and starts timing when it determines that a certain image frame meets the action determination start condition. In this way, the computer device continues to acquire image frames while timing, and determines whether the image frame meets the action determination end condition. Only when the timing duration does not reach the preset duration and the image frames that continue to be acquired meet the action determination end condition, is it determined that the action of triggering the addition of additional elements is detected. If the computer device does not detect that the image frames that continue to be acquired within the timing time period meet the action determination end condition until the timing duration reaches the preset duration, it is determined that the action of triggering the addition of additional elements is not detected. At this time, the computer device continues to acquire image frames, no longer determines whether the image frame meets the action determination end condition, but determines whether the image frame meets the action determination start condition, so as to continue to start timing when it determines that a certain image frame meets the action determination start condition, so as to continue to detect the action of triggering the addition of additional elements. Among them, the preset duration is the duration of the action formation determined based on actual experience.

[0100] In one embodiment, the image processing method also includes: when the proportion of the intersection of the target area and the reference area in the acquired image frame exceeds a first preset value, it is determined that the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition; or, when the proportion of the intersection of the target area and the reference area in the acquired image frame exceeds a second preset value, and the center position of the target area is above the center position of the reference area, it is determined that the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition.

[0101] Among them, the first preset value and the second preset value are pre-set values. The first preset value can be specifically 0.5, and the second preset value can be specifically 0.2. It can be understood that the intersection of the target area and the reference area in the acquired image frame accounts for a proportion of the target area that exceeds the first preset value, or the intersection of the target area and the reference area in the acquired image frame accounts for a proportion of the target area that exceeds the second preset value, and the center position of the target area is located above the center position of the reference area. This is the positional relationship between the target area and the reference area when the action determination start condition is met based on actual experience.

[0102] Figure 5 Schematic diagram of an image frame that meets the action determination start condition in one embodiment. In this embodiment, the action that triggers the addition of additional elements is the head-lifting action, the target area is the hand area, and the reference area is the face area. Figure 5 (a), it can be seen that the intersection of the hand area and the face area accounts for a proportion of the hand area that exceeds the first preset value (0.5), and it can be determined that the positional relationship between the target area and the reference area in the image frame meets the action determination start condition. Figure 5 (b) It can be seen that when the intersection of the hand area and the facial area accounts for a proportion of the hand area that exceeds the second preset value (0.2), and the center position O1 of the hand area is located above the center position O2 of the facial area, it can be determined that the positional relationship between the target area and the reference area in the image frame meets the action determination start condition.

[0103] In one embodiment, the image processing method also includes: determining a reference position in a reference area in an image frame that continues to be acquired; when a target area in an image frame that continues to be acquired is located above the reference position in the reference area, determining that a positional relationship between the target area and the reference area in the image frame that continues to be acquired satisfies an action determination termination condition.

[0104] The reference position is a comparison position used to determine whether the positional relationship between the target area and the reference area in the image frame meets the action determination end condition. It can be understood that the target area in the image frame is located above the reference position in the reference area, which is the positional relationship between the target area and the reference area when the action determination end condition is met based on actual experience.

[0105] Figure 6 Schematic diagram of an image frame that meets the action determination end condition in one embodiment. In this embodiment, the action that triggers the addition of additional elements is the head-lifting action, the target area is the hand area, the reference area is the face area, and the reference position is the position of the eyebrows in the face area. Figure 6 , it can be seen that the hand area in the image frame is located above the eyebrow position in the facial area, and it can be determined that the positional relationship between the target area and the reference area in the image frame meets the action determination end condition.

[0106] In the above embodiment, a basis for specifically judging whether an image frame satisfies the action determination start condition or the action determination end condition is provided, thereby ensuring that the action determination is carried out effectively. Moreover, only when it is judged that the action determination start condition is satisfied and the action determination end condition is continuously satisfied within a preset time period, it is judged that an action is detected, so that the action detection is in line with actual cognition and effective.

[0107] It can be understood that determining that the action determination start condition is met and continuing to determine that the action determination end condition is met within the preset time length means that the time interval between the acquisition time of the image frame that meets the action determination start condition and the acquisition time of the image frame that meets the action determination end condition is less than or equal to the preset time length.

[0108] In one embodiment, the target area is a hand area; the reference area is a facial area; and the reference position is a position of eyebrows. Adding the additional element to an image frame acquired after an image frame acquired at a later time includes: determining, in an image frame acquired after the continuously acquired image frame, an area formed by a boundary between a position of eyebrows in the facial area and a position of eyebrows in the hand area; and adaptively adding the additional element to the determined area in an image frame acquired after the continuously acquired image frame.

[0109] Specifically, the computer device can perform face detection on the image frames collected after the continued acquisition of image frames, determine the left and right eyebrow reference points in the face area of ​​the image frame, determine the position of the eyebrows based on the reference points, and then determine the area formed by the boundary between the position of the eyebrows in the facial area and the position of the eyebrows in the hand area, thereby adaptively adding additional elements to the determined area.

[0110] Among them, adaptively adding the additional element to the determined area may be to adjust the size of the additional element to the size of the determined area, so that the size of the determined area will gradually increase with the movement of the hand area, and the additional element will also gradually increase in display size as the size of the determined area increases. Adaptively adding the additional element to the determined area may be to add a partial area of ​​the additional element to the determined area, wherein the partial area of ​​the additional element is bounded by a certain boundary of the additional element, which corresponds to the boundary of the determined area, so that the size of the determined area will gradually increase with the movement of the hand area, and the additional element will also gradually change from partial display to full display as the size of the determined area increases, and the displayed part becomes larger and larger.

[0111] Figure 7 Schematic diagram of an image frame with additional elements added in one embodiment. In this embodiment, the action that triggers the addition of additional elements is the head-lifting action, the target area is the hand area, the reference area is the face area, and the reference position is the position of the eyebrows in the face area. Figure 7 , it can be seen that from (a) to (b), the area formed by the boundary between the eyebrows in the facial area and the hand area near the eyebrows gradually increases, and the size of the additional elements added in this area also becomes larger and larger.

[0112] In this embodiment, additional elements are adaptively added to the determined area until they are completely added, rather than directly adding the complete additional elements, which avoids the monotony and abruptness of the additional element adding process, and enhances interactivity by gradually and adaptively adding additional elements according to the movement of the hand area.

[0113] In another embodiment, the computer device may further blur the border of the additional element.

[0114] In one embodiment, the image processing method also includes: when the action triggering the addition of additional elements is not detected, playing the acquired image frames frame by frame according to the acquisition sequence; after the action triggering the addition of additional elements is detected, playing the image frames after the additional elements are added frame by frame according to the acquisition sequence.

[0115] Specifically, the computer device can play the captured image frames in real time after capturing the image frames. When the action of triggering the addition of additional elements is not detected, that is, the currently captured image frames do not need to add additional elements, then the acquired image frames can be directly rendered to form a preview screen to display the captured image frames in a visual way. After the action of triggering the addition of additional elements is detected, that is, the currently captured image frames need to add additional elements, then the additional elements are added to the image frames captured after the action of triggering the addition of additional elements is detected, and the image frames after the additional elements are added are rendered to form a preview screen to display the image frames after the additional elements are added in a visual way.

[0116] In this embodiment, during the shooting process, the action of adding additional elements is triggered by detecting the captured image frame, and a preview screen is generated in real time according to the captured video frame or the image frame after adding the additional elements for the user to watch. In this way, the user can understand the content of the recorded video in real time, so as to correct or re-record in time when an error occurs.

[0117] In one embodiment, the image processing method also includes: replacing the corresponding image frame before adding the additional element with the image frame after adding the additional element; according to the acquisition time of the image frame determined after the replacement, generating a recorded video according to the timing of the acquisition time of the image frame determined after the replacement; wherein the acquisition time of the image frame obtained by replacement in the image frame determined after the replacement is the acquisition time of the corresponding image frame to be replaced.

[0118] The image frame determined after replacement includes the image frame originally captured before the additional element is added, and also includes the image frame obtained by adding the additional element after the additional element is added. The acquisition time of the image frame originally captured without the replacement operation in the image frame determined after replacement is the real acquisition time of the image frame. The acquisition time of the image frame obtained by replacement in the image frame determined after replacement is the acquisition time of the corresponding image frame replaced.

[0119] For example, the original captured image frames are A, B, C, and D, and additional elements are added starting from image frame C. Image frame C1 is obtained by adding elements to image frame C, and image frame D1 is obtained by adding elements to image frame D. Then, image frame C is replaced by image frame C1, and image frame D is replaced by image frame D1. The image frames determined after the replacement are A, B, C1, and D1, that is, these image frames are used to generate the video.

[0120] Specifically, the computer device may replace the corresponding image frame before the additional element is added with the image frame obtained after the additional element is added, and then generate the recorded video according to the time sequence of the image frames determined after the replacement according to the acquisition time of each image frame determined after the replacement. The time sequence according to the acquisition time may be in reverse order of time or in order of time.

[0121] Furthermore, after generating the recorded video, the computer device may share the video in a social session or publish the video to a social publishing platform.

[0122] In this embodiment, the captured image frames are automatically and in real time processed during the shooting process, and the video is generated in real time, avoiding the cumbersome steps caused by subsequent manual processing, greatly simplifying the operation, and improving the efficiency of video generation.

[0123] Figure 8 A flowchart of an image processing method in a specific embodiment is shown. In this embodiment, the action that triggers the addition of additional elements is a head-lifting action, the target area is the hand area, the reference area is the facial area, and the reference position is the position of the eyebrows in the facial area. The computer device is a terminal. A video recording application is installed on the terminal. The terminal can run the video recording application according to user instructions, call the built-in camera of the terminal through the video recording application to capture image frames, and when capturing image frames, obtain the captured image frames in real time according to the capture timing of the image frames.

[0124] After acquiring the captured image frame, the terminal can determine the gesture type corresponding to the hand area in the acquired image frame, and determine whether the gesture type is a trigger type; if not, acquire the next image frame, and continue to determine the gesture type corresponding to the hand area in the image frame; if so, determine whether the positional relationship between the target area and the reference area in the image frame meets the action determination start condition. The action determination start condition is: the intersection of the target area and the reference area in the image frame accounts for a proportion of the target area that exceeds a first preset value, or the intersection of the target area and the reference area in the image frame accounts for a proportion of the target area that exceeds a second preset value, and the center position of the target area is located above the center position of the reference area.

[0125] When the terminal determines that the image frame does not meet the action determination start condition, it acquires the next image frame and continues to detect and determine the gesture type corresponding to the hand area in the image frame; when it determines that the image frame meets the action determination start condition, it starts timing and continues to acquire the next image frame. The terminal then determines the gesture type corresponding to the hand area in the acquired image frame and determines whether the gesture type is a trigger type; if not, it acquires the next image frame and continues to detect and determine the gesture type corresponding to the hand area in the image frame; if so, it determines whether the positional relationship between the target area and the reference area in the image frame meets the action determination end condition. Among them, the action determination end condition is: the target area in the image frame is located above the reference position in the reference area.

[0126] When the terminal detects that the image frame meets the action determination end condition before the timing duration reaches the preset duration, it determines that the action that triggers the addition of the additional element is detected, and in the image frames collected after the image frames that continue to be acquired, the area formed by the boundary between the position of the eyebrows in the facial area and the position of the eyebrows in the hand area close to the position of the eyebrows is determined; in the image frames collected after the image frames that continue to be acquired, the additional element is adaptively added to the determined area. When the terminal still does not detect that the image frame meets the action determination end condition after the timing duration reaches the preset duration, the next image frame is acquired, and the gesture type corresponding to the hand area in the image frame is continuously determined, and when the gesture type is a trigger type, the action determination start condition is detected.

[0127] The terminal may also, during image processing, replace the corresponding image frames before adding the additional elements with the image frames after adding the additional elements in real time, and generate a recorded video with the image frames determined after the replacement according to the acquisition time of the image frames determined after the replacement in the sequence of the acquisition time; or after completing the image frame acquisition, generate a recorded video with the image frames determined after the replacement in the sequence of the acquisition time according to the acquisition time of the image frames determined after the replacement.

[0128] It should be understood that, although each step in the flow chart of the above-mentioned embodiments is shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a part of the steps in the above-mentioned embodiments may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0129] like Fig. 9 As shown, in one embodiment, an image processing device 900 is provided. Fig. 9 The image processing device 900 includes: an acquisition module 901, a determination module 902, a judgment module 903 and an adding module 904.

[0130] The acquisition module 901 is used to acquire the captured image frames.

[0131] The determination module 902 is used to determine the target area and the reference area obtained by image semantic segmentation in the acquired image frame.

[0132] The determination module 903 is used to determine that an action triggering the addition of an additional element is detected when the positional relationship between the target area and the reference area in the image frame with an earlier acquisition time meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame with a later acquisition time meets the action determination end condition.

[0133] The adding module 904 is used to obtain additional elements when a motion is detected; and add the additional elements to the image frame acquired after the image frame acquired at a later time.

[0134] In one embodiment, the determination module 902 is also used to input the acquired image frame into the image semantic segmentation model; output the target area probability distribution matrix and the reference area probability distribution matrix through the image semantic segmentation model; determine the target area in the acquired image frame according to the target area probability distribution matrix; determine the reference area in the acquired image frame according to the reference area probability distribution matrix.

[0135] In one embodiment, the target area is the hand area; the reference area is the face area. The determination module 902 is also used to determine the gesture type corresponding to the hand area in the acquired image frame. The judgment module 903 is also used to determine that the action of triggering the addition of additional elements is detected when the positional relationship between the hand area and the face area in the image frame with the previous acquisition time meets the action judgment start condition and the positional relationship between the hand area and the face area in the image frame with the later acquisition time meets the action judgment end condition.

[0136] In one embodiment, the determination module 903 is also used to start timing when the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition; when the timing duration does not reach the preset duration and the positional relationship between the target area and the reference area in the continuously acquired image frames meets the action determination end condition, it is determined that an action that triggers the addition of additional elements is detected; wherein the acquired image frame is an image frame with an earlier acquisition time, and the continuously acquired image frame is an image frame with a later acquisition time.

[0137] In one embodiment, the determination module 903 is also used to determine that the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition when the proportion of the intersection of the target area and the reference area in the acquired image frame exceeds a first preset value; or, determine that the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition when the proportion of the intersection of the target area and the reference area in the acquired image frame exceeds a second preset value and the center position of the target area is above the center position of the reference area.

[0138] In one embodiment, the determination module 903 is also used to determine a reference position in a reference area in an image frame that is continuously acquired; when a target area in an image frame that is continuously acquired is located above the reference position in the reference area, it is determined that the positional relationship between the target area and the reference area in the image frame that is continuously acquired satisfies an action determination termination condition.

[0139] In one embodiment, the target area is the hand area; the reference area is the face area; and the reference position is the position of the eyebrows. The adding module 904 is further used to determine, in the image frames acquired after the continuously acquired image frames, an area formed by the boundary between the position of the eyebrows in the face area and the position of the eyebrows in the hand area; and in the image frames acquired after the continuously acquired image frames, adaptively add the additional element to the determined area.

[0140] In one embodiment, the acquisition module 901 is further used to play the acquired image frames frame by frame according to the acquisition time sequence when no action triggering the addition of additional elements is detected. The adding module 904 is further used to play the image frames after the additional elements are added frame by frame according to the acquisition time sequence after the action triggering the addition of additional elements is detected.

[0141] In one embodiment, the adding module 904 is also used to replace the corresponding image frame before adding the additional element with the image frame after adding the additional element; according to the acquisition time of the image frame determined after the replacement, the image frame determined after the replacement is used to generate a recorded video according to the timing of the acquisition time; wherein the acquisition time of the image frame obtained by replacement in the image frame determined after the replacement is the acquisition time of the corresponding image frame to be replaced.

[0142] Fig.10 The internal structure diagram of a computer device in one embodiment is shown. The computer device may specifically be Figure 1 The terminal 110 or the server 120 in FIG. Fig.10 As shown, the computer device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the image processing method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement the image processing method. Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, the image processing apparatus provided by the present application can be implemented in the form of a computer program. The computer program can be Fig.10 The non-volatile storage medium of the computer device can store various program modules constituting the image processing apparatus, for example, Fig. 9The acquisition module 901, determination module 902, judgment module 903 and adding module 904 shown in the figure are shown in the figure. The computer program composed of various program modules enables the processor to execute the steps of the image processing method of each embodiment of the present application described in this specification.

[0144] For example, Fig.10 The computer device shown can be Fig. 9 The acquisition module 901 in the image processing device 900 shown acquires the acquired image frame. The determination module 902 determines the target area and the reference area obtained by image semantic segmentation in the acquired image frame. The determination module 903 determines that the action triggering the addition of additional elements is detected when the positional relationship between the target area and the reference area in the image frame with the earlier acquisition time meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame with the later acquisition time meets the action determination end condition. The adding module 904 acquires the additional elements when the action is detected; and adds the additional elements to the image frame acquired after the image frame with the later acquisition time.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor performs the following steps: acquiring an acquired image frame; determining a target area and a reference area obtained by image semantic segmentation in the acquired image frame; when the positional relationship between the target area and the reference area in the image frame with an earlier acquisition time satisfies an action determination start condition, and the positional relationship between the target area and the reference area in the image frame with a later acquisition time satisfies an action determination end condition, determining that an action that triggers the addition of an additional element is detected; acquiring the additional element when the action is detected; and adding the additional element to the image frame acquired after the image frame with a later acquisition time.

[0146] In one embodiment, a target area and a reference area obtained by image semantic segmentation are determined in an acquired image frame, including: inputting the acquired image frame into an image semantic segmentation model; outputting a target area probability distribution matrix and a reference area probability distribution matrix through the image semantic segmentation model; determining the target area in the acquired image frame according to the target area probability distribution matrix; and determining the reference area in the acquired image frame according to the reference area probability distribution matrix.

[0147] In one embodiment, the target area is a hand area; the reference area is a facial area. The computer program further causes the processor to perform the following steps: determine the gesture type corresponding to the hand area in the acquired image frame; when the gesture type is a trigger type, when the positional relationship between the target area and the reference area in the image frame acquired at the previous time satisfies the action determination start condition, and the positional relationship between the target area and the reference area in the image frame acquired at the later time satisfies the action determination end condition, then determine that the action of triggering the addition of additional elements is detected, including: when the positional relationship between the hand area and the facial area in the image frame acquired at the previous time satisfies the action determination start condition, and the positional relationship between the hand area and the facial area in the image frame acquired at the later time satisfies the action determination end condition, then determine that the action of triggering the addition of additional elements is detected.

[0148] In one embodiment, when the positional relationship between the target area and the reference area in the image frame with an earlier acquisition time meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame with a later acquisition time meets the action determination end condition, it is determined that the action of triggering the addition of additional elements is detected, including: starting timing from the time when the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition; when the timing duration does not reach a preset duration and the positional relationship between the target area and the reference area in the image frames that continue to be acquired meets the action determination end condition, it is determined that the action of triggering the addition of additional elements is detected; wherein the acquired image frame is the image frame with an earlier acquisition time, and the image frame that continues to be acquired is the image frame with a later acquisition time.

[0149] In one embodiment, the computer program further causes the processor to perform the following steps: when the proportion of the intersection of the target area and the reference area in the acquired image frame to the target area exceeds a first preset value, determining that the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition; or, when the proportion of the intersection of the target area and the reference area in the acquired image frame to the target area exceeds a second preset value and the center position of the target area is above the center position of the reference area, determining that the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition.

[0150] In one embodiment, the computer program also causes the processor to perform the following steps: determining a reference position in a reference area in an image frame that continues to be acquired; and when a target area in an image frame that continues to be acquired is located above the reference position in the reference area, determining that a positional relationship between the target area and the reference area in the image frame that continues to be acquired satisfies an action determination termination condition.

[0151] In one embodiment, the target area is a hand area; the reference area is a facial area; and the reference position is a position of eyebrows. Adding the additional element to an image frame acquired after an image frame acquired at a later time includes: determining, in an image frame acquired after the continuously acquired image frame, an area formed by a boundary between a position of eyebrows in the facial area and a position of eyebrows in the hand area; and adaptively adding the additional element to the determined area in an image frame acquired after the continuously acquired image frame.

[0152] In one embodiment, the computer program further causes the processor to perform the following steps: when no action is detected that triggers the addition of additional elements, the acquired image frames are played frame by frame in the order of acquisition; after the action is detected that triggers the addition of additional elements, the image frames after the additional elements are added are played frame by frame in the order of acquisition.

[0153] In one embodiment, the computer program further causes the processor to perform the following steps: replacing the corresponding image frame before adding the additional element with the image frame after adding the additional element; generating a recorded video according to the acquisition time of the image frame determined after the replacement according to the sequence of the acquisition time; wherein the acquisition time of the image frame obtained by replacement in the image frame determined after the replacement is the acquisition time of the corresponding image frame to be replaced.

[0154] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the following steps: acquiring a captured image frame; determining a target area and a reference area obtained by image semantic segmentation in the acquired image frame; when the positional relationship between the target area and the reference area in the image frame with an earlier capture time satisfies an action determination start condition, and when the positional relationship between the target area and the reference area in the image frame with a later capture time satisfies an action determination end condition, determining that an action that triggers the addition of an additional element is detected; acquiring the additional element when the action is detected; and adding the additional element to the image frame captured after the image frame with a later capture time.

[0155] In one embodiment, a target area and a reference area obtained by image semantic segmentation are determined in an acquired image frame, including: inputting the acquired image frame into an image semantic segmentation model; outputting a target area probability distribution matrix and a reference area probability distribution matrix through the image semantic segmentation model; determining the target area in the acquired image frame according to the target area probability distribution matrix; and determining the reference area in the acquired image frame according to the reference area probability distribution matrix.

[0156] In one embodiment, the target area is a hand area; the reference area is a facial area. The computer program further causes the processor to perform the following steps: determine the gesture type corresponding to the hand area in the acquired image frame; when the gesture type is a trigger type, when the positional relationship between the target area and the reference area in the image frame acquired at the previous time satisfies the action determination start condition, and the positional relationship between the target area and the reference area in the image frame acquired at the later time satisfies the action determination end condition, then determine that the action of triggering the addition of additional elements is detected, including: when the positional relationship between the hand area and the facial area in the image frame acquired at the previous time satisfies the action determination start condition, and the positional relationship between the hand area and the facial area in the image frame acquired at the later time satisfies the action determination end condition, then determine that the action of triggering the addition of additional elements is detected.

[0157] In one embodiment, when the positional relationship between the target area and the reference area in the image frame with an earlier acquisition time meets the action determination start condition, and the positional relationship between the target area and the reference area in the image frame with a later acquisition time meets the action determination end condition, it is determined that the action of triggering the addition of additional elements is detected, including: starting timing from the time when the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition; when the timing duration does not reach a preset duration and the positional relationship between the target area and the reference area in the image frames that continue to be acquired meets the action determination end condition, it is determined that the action of triggering the addition of additional elements is detected; wherein the acquired image frame is the image frame with an earlier acquisition time, and the image frame that continues to be acquired is the image frame with a later acquisition time.

[0158] In one embodiment, the computer program further causes the processor to perform the following steps: when the proportion of the intersection of the target area and the reference area in the acquired image frame to the target area exceeds a first preset value, determining that the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition; or, when the proportion of the intersection of the target area and the reference area in the acquired image frame to the target area exceeds a second preset value and the center position of the target area is above the center position of the reference area, determining that the positional relationship between the target area and the reference area in the acquired image frame meets the action determination start condition.

[0159] In one embodiment, the computer program also causes the processor to perform the following steps: determining a reference position in a reference area in an image frame that continues to be acquired; and when a target area in an image frame that continues to be acquired is located above the reference position in the reference area, determining that a positional relationship between the target area and the reference area in the image frame that continues to be acquired satisfies an action determination termination condition.

[0160] In one embodiment, the target area is a hand area; the reference area is a facial area; and the reference position is a position of eyebrows. Adding the additional element to an image frame acquired after an image frame acquired at a later time includes: determining, in an image frame acquired after the continuously acquired image frame, an area formed by a boundary between a position of eyebrows in the facial area and a position of eyebrows in the hand area; and adaptively adding the additional element to the determined area in an image frame acquired after the continuously acquired image frame.

[0161] In one embodiment, the computer program further causes the processor to perform the following steps: when no action is detected that triggers the addition of additional elements, the acquired image frames are played frame by frame in the order of acquisition; after the action is detected that triggers the addition of additional elements, the image frames after the additional elements are added are played frame by frame in the order of acquisition.

[0162] In one embodiment, the computer program further causes the processor to perform the following steps: replacing the corresponding image frame before adding the additional element with the image frame after adding the additional element; generating a recorded video according to the acquisition time of the image frame determined after the replacement according to the sequence of the acquisition time; wherein the acquisition time of the image frame obtained by replacement in the image frame determined after the replacement is the acquisition time of the corresponding image frame to be replaced.

[0163] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. An image processing method, comprising: Get the captured image frame; Determine a hand region and a face region obtained by image semantic segmentation in the acquired image frame; Determine the gesture type corresponding to the hand region in the acquired image frame; When the gesture type is a trigger type, when the positional relationship between the hand area and the face area in the image frame acquired at the previous time satisfies the action determination start condition, and the positional relationship between the hand area and the face area in the image frame acquired at the later time satisfies the action determination end condition, it is determined that the action of triggering the addition of the additional element is detected; acquiring additional elements upon detecting said action; The additional element is added to an image frame acquired after the image frame acquired at a later time.

2. The method according to claim 1, characterized in that The step of determining the hand region and the facial region obtained by image semantic segmentation in the acquired image frame comprises: Input the acquired image frames into the image semantic segmentation model; Outputting a hand region probability distribution matrix and a facial region probability distribution matrix through the image semantic segmentation model; Determining a hand region in an acquired image frame according to the hand region probability distribution matrix; The facial region in the acquired image frame is determined according to the facial region probability distribution matrix.

3. The method according to claim 1, characterized in that When the positional relationship between the hand area and the face area in the image frame acquired earlier meets the action determination start condition, and the positional relationship between the hand area and the face area in the image frame acquired later meets the action determination end condition, then determining that the action of triggering the addition of the additional element is detected includes: The timing starts when the positional relationship between the hand area and the face area in the acquired image frame meets the action determination start condition; When the timing duration does not reach the preset duration and the positional relationship between the hand area and the face area in the image frames that continue to be acquired meets the action determination end condition, it is determined that the action of triggering the addition of the additional element is detected; The acquired image frames are image frames acquired earlier in time, and the image frames that continue to be acquired are image frames acquired later in time.

4. The method according to claim 3, characterized in that The method further comprises: When the proportion of the intersection of the hand area and the facial area in the acquired image frame to the hand area exceeds a first preset value, it is determined that the positional relationship between the hand area and the facial area in the acquired image frame meets the action determination start condition.

5. The method according to claim 3, characterized in that: The method further comprises: When the proportion of the intersection of the hand area and the facial area in the acquired image frame exceeds a second preset value and the center position of the hand area is located above the center position of the facial area, it is determined that the positional relationship between the hand area and the facial area in the acquired image frame meets the action determination start condition.

6. The method according to claim 3, characterized in that The method further comprises: determining the location of eyebrows in the facial region in the continuously acquired image frames; When the hand region in the continuously acquired image frame is located above the position of the eyebrows in the facial region, it is determined that the positional relationship between the hand region and the facial region in the continuously acquired image frame satisfies the action determination end condition.

7. The method according to claim 6, characterized in that The adding the additional element to the image frame acquired after the image frame acquired at a later time comprises: In the image frames acquired after the continuously acquired image frames, determining an area formed by a boundary between a position where the eyebrows are located in the facial region and a position where the eyebrows are located in the hand region; In the image frame acquired after the continuously acquired image frame, the additional element is adaptively added to the determined area.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When no action is detected to trigger the addition of the additional element, the acquired image frames are played frame by frame according to the acquisition time sequence; After the action of triggering the addition of the additional element is detected, the image frames after the additional element is added are played frame by frame according to the acquisition time sequence.

9. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: replacing the corresponding image frame before adding the additional element with the image frame after adding the additional element; According to the acquisition time of the image frame determined after the replacement, the image frame determined after the replacement is generated into a recorded video according to the time sequence of the acquisition time; The acquisition time of the image frame obtained by replacement in the image frame determined after replacement is the acquisition time of the corresponding image frame being replaced.

10. An image processing device, comprising: An acquisition module, used for acquiring the collected image frames; A determination module, used to determine the hand area and the facial area obtained by image semantic segmentation in the acquired image frame; determine the gesture type corresponding to the hand area in the acquired image frame; A determination module, for determining that, when the gesture type is a trigger type, the action of triggering the addition of an additional element is detected when the positional relationship between the hand area and the face area in the image frame acquired earlier meets the action determination start condition and the positional relationship between the hand area and the face area in the image frame acquired later meets the action determination end condition; An adding module for obtaining additional elements when the action is detected; The additional element is added to an image frame acquired after the image frame acquired at a later time.

11. The device according to claim 10, characterized in that The determination module is also used to input the acquired image frame into an image semantic segmentation model; determine the hand area in the acquired image frame according to the hand area probability distribution matrix and the face area probability distribution matrix output by the image semantic segmentation model; and determine the face area in the acquired image frame according to the face area probability distribution matrix.

12. The device according to claim 10, characterized in that The determination module is also used to start timing when the positional relationship between the hand area and the facial area in the acquired image frame meets the action determination start condition; when the timing duration does not reach the preset duration and the positional relationship between the hand area and the facial area in the continuously acquired image frames meets the action determination end condition, it is determined that the action of triggering the addition of additional elements is detected; wherein the acquired image frame is the image frame with an earlier acquisition time, and the continuously acquired image frame is the image frame with a later acquisition time.

13. The device according to claim 12, characterized in that The determination module is also used to determine that the positional relationship between the hand area and the face area in the acquired image frame meets the action determination start condition when the proportion of the intersection of the hand area and the face area in the acquired image frame exceeds a first preset value.

14. The device according to claim 12, characterized in that The determination module is also used to determine that the positional relationship between the hand area and the facial area in the acquired image frame meets the action determination start condition when the proportion of the intersection of the hand area and the facial area in the acquired image frame exceeds a second preset value and the center position of the hand area is located above the center position of the facial area.

15. The device according to claim 12, characterized in that The determination module is also used to determine the position of the eyebrows in the facial area of ​​the image frame that is continuously acquired; when the hand area in the image frame that is continuously acquired is located above the position of the eyebrows in the facial area, it is determined that the positional relationship between the hand area and the facial area in the image frame that is continuously acquired meets the action determination end condition.

16. The device according to claim 15, characterized in that The adding module is also used to determine, in the image frames collected after the continuously acquired image frames, an area formed by the boundary between the eyebrow position in the facial area and the hand area close to the eyebrow position; in the image frames collected after the continuously acquired image frames, adaptively add the additional element to the determined area.

17. The device according to any one of claims 10 to 16, characterized in that The acquisition module is also used to play the acquired image frames frame by frame according to the acquisition sequence when no action triggering the addition of additional elements is detected; after the action triggering the addition of additional elements is detected, the image frames after the additional elements are added are played frame by frame according to the acquisition sequence.

18. The device according to any one of claims 10 to 16, characterized in that The adding module is also used to replace the corresponding image frame before adding the additional element with the image frame after adding the additional element; according to the acquisition time of the image frame determined after the replacement, the image frame determined after the replacement is used to generate a recorded video according to the time sequence of the acquisition time; wherein the acquisition time of the image frame obtained by replacement in the image frame determined after the replacement is the acquisition time of the corresponding image frame to be replaced.

19. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is caused to execute the steps of the method according to any one of claims 1 to 9.

20. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • System for implementing remote control interaction in virtual three-dimensional scene

    CN101465957A