Image processing methods and apparatuses, electronic devices and storage media

Abstract

An image processing method, image processing apparatus, electronic device, and storage medium are disclosed. The image processing method includes: capturing interactive actions performed by an interactive object on a recognition object using an imaging device, obtaining multiple frame images; recognizing the interactive actions of the interactive object based on the multiple frame images; blurring the multiple frame images to obtain multiple processed images corresponding to each frame image, wherein the blurring ensures that the interactive object does not obstruct the displayed content of the recognition object in each processed image; and executing a corresponding interactive operation based on the multiple processed images and the interactive actions. This image processing method provides a form of air-based interactive operation with the recognition object, rather than being limited to interaction only through I / O interfaces, thus improving intelligence; and it can accurately recognize the interactive actions of the interactive object, accurately recognizing the content in the interactive area when the interactive action is a selection action, thereby improving recognition accuracy.

Description

Technical Field

[0001] Embodiments of this disclosure relate to an image processing method, an image processing apparatus, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in identifying, tracking, and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting "information" from images or multidimensional data. Computer vision can also be seen as the science of how to enable artificial systems to "perceive" from images or multidimensional data. Summary of the Invention

[0003] This disclosure provides at least one embodiment of an image processing method, comprising: capturing interactive actions performed by an interactive action object on a recognition object using a shooting device to obtain multiple frame images; performing a blurring process on the multiple frame images to obtain multiple processed images corresponding to the multiple frame images, wherein the blurring process ensures that in each processed image, the interactive action object does not obstruct the displayed content in the recognition object; recognizing the interactive action of the interactive action object based on the multiple frame images; and performing a corresponding interactive operation based on the multiple processed images and the interactive action.

[0004] For example, in at least one embodiment of the image processing method provided in this disclosure, the plurality of frame images are blurred to obtain a plurality of processed images corresponding to the plurality of frame images, including: determining a reference image corresponding to the recognition object, wherein the reference image includes at least a portion of the display content corresponding to the recognition object; for each frame image: using a segmentation model to determine an occlusion region in the frame image composed of at least a portion of the interactive action object, wherein the at least a portion of the display content includes the display content in the occlusion region; merging the reference image and the occlusion region to obtain a first intermediate image corresponding to the frame image; blurring the occlusion region in the first intermediate image to obtain the processed image corresponding to the frame image.

[0005] For example, in at least one embodiment of the present disclosure, an image processing method is provided to determine a reference image corresponding to the identified object, including: capturing an image that includes all the display content of the identified object as the reference image.

[0006] For example, in at least one embodiment of the image processing method provided in this disclosure, determining the reference image corresponding to the identified object includes: sequentially performing background merging processing on the plurality of frame images to obtain the reference image.

[0007] For example, in at least one embodiment of the image processing method provided in this disclosure, background merging processing is sequentially performed on the plurality of frame images to obtain the reference image, including: for the current frame image in which the background merging processing is performed, using the segmentation model to determine the occlusion region in the current frame image that is at least partly composed of the interactive action object; marking the pixels in the occlusion region of the current frame image to obtain a second intermediate image corresponding to the frame image; obtaining a historical background merged image, wherein, in response to the current frame image being the first image to undergo the background merging processing, the second intermediate image corresponding to the current frame image is used as the historical background merged image; performing matching processing on the historical background merged image and the second intermediate image, in response to the result of the matching processing indicating that the identified object has not moved, performing background fusion on the second intermediate image and the historical background merged image to obtain a fused image, and in response to the proportion of pixels marked as occlusion positions in the fused image being less than a proportion threshold, determining the fused image as the reference image, otherwise updating the historical background merged image to the fused image. The background merging process is then performed on the next frame image. In response to the result of the matching process indicating that the identified object has moved, the historical background merged image is updated to the second intermediate image, and the background merging process is then performed on the next frame image.

[0008] For example, in at least one embodiment of the image processing method provided in this disclosure, the matching process between the historical background merged image and the second intermediate image includes: determining a first histogram vector corresponding to the historical background merged image and a second histogram vector corresponding to the second intermediate image; comparing the similarity between the first histogram vector and the second histogram vector; and determining the result of the matching process based on the similarity.

[0009] For example, in at least one embodiment of the image processing method provided in this disclosure, comparing the similarity between the first histogram vector and the second histogram vector, and determining the result of the matching process based on the similarity, includes: calculating the correlation coefficient between the first histogram vector and the second histogram vector; calculating the Euclidean distance between the first histogram vector and the second histogram vector; calculating the weighted average of the correlation coefficient and the Euclidean distance as the similarity; and determining that the result of the matching process indicates that the identified object has not moved if the similarity is greater than a similarity threshold, and determining that the result of the matching process indicates that the identified object has moved if the similarity is less than the similarity threshold.

[0010] For example, in at least one embodiment of the image processing method provided in this disclosure, the second intermediate image and the historical background merged image are subjected to background fusion to obtain a fused image, including: determining one or more pixels in the historical background merged image that are marked as occlusion positions; determining pixels in the second intermediate image that correspond to the one or more pixels as replacement pixels; determining at least one pixel in the replacement pixels that is not marked as an occlusion position; and replacing the value of the pixel in the one or more pixels that corresponds to the value of the at least one pixel with the value of the at least one pixel to obtain the fused image.

[0011] For example, in at least one embodiment of the image processing method provided in this disclosure, merging the reference image and the occluded region to obtain a first intermediate image corresponding to the frame image includes: adding display content corresponding to the occluded region in the frame image to the reference image to obtain the first intermediate image corresponding to the frame image.

[0012] For example, in at least one embodiment of the image processing method provided in this disclosure, blurring the occluded area in the first intermediate image to obtain the processed image corresponding to the frame image includes: increasing the transparency of the occluded area in the first intermediate image to obtain the processed image corresponding to the frame image.

[0013] For example, in at least one embodiment of the present disclosure, an image processing method is provided to identify the interactive action of the interactive action object based on the plurality of frame images, including: for each frame image, determining the position of the key points of the interactive action object in the frame image; and identifying the interactive action based on the position of the key points of the interactive action object in the plurality of frame images.

[0014] For example, in at least one embodiment of the image processing method provided in this disclosure, the interactive action object is a gesture action object. For each frame image, determining the position of the key points of the interactive action object in the frame image includes: using a key point detection model to identify target key points in the gesture action object; extracting a detection region in the frame image including the target key points based on the target key points; extracting at least one corner feature position in the detection region; performing Gaussian convolution calculation at each corner feature position to obtain a corner feature map corresponding to the frame image; performing a two-dimensional convolution calculation between the corner feature map and the corner feature map corresponding to a reference frame image to obtain a calculation result, wherein the reference frame image is a frame image that is located before and adjacent to the frame image in the shooting order among the plurality of frame images; determining the position corresponding to the maximum value in the calculation result; and correcting the position of the target key point based on the position corresponding to the maximum value to obtain the position of the key points of the gesture action object.

[0015] For example, in at least one embodiment of the image processing method provided in this disclosure, the interactive action object includes a first hand and a second hand, and the key points of the interactive action object include a first key point corresponding to the first hand and a second key point corresponding to the second hand. The interactive action is identified based on the position of the key points of the interactive action object in the plurality of frame images, including: determining the interactive action as a first selection action in response to the duration of the position changes of the first key point and the second key point being less than a position threshold being greater than a time threshold; and determining an interactive area based on the position of the first key point and the second key point.

[0016] For example, in at least one embodiment of the image processing method provided in this disclosure, determining an interactive region based on the positions of the first key point and the second key point includes: determining an outer region box based on the positions of the first key point and the second key point, and using the outer region box as the interactive region.

[0017] For example, in at least one embodiment of the image processing method provided in this disclosure, a corresponding interactive operation is performed based on the plurality of processed images and the interactive action, including: in response to the interactive action being the first selection action, obtaining a text description of the content in the interactive area based on the plurality of processed images.

[0018] For example, in at least one embodiment of the present disclosure, an image processing method is provided to identify the interactive action based on the position of the key points of the interactive action object in the plurality of frame images, including: in response to the key points of the interactive action object moving on the identified object and detecting a stop action: determining the interactive action as a second selected action, and determining the movement trajectory of the key points of the interactive action object between the stop actions based on the position of the key points of the interactive action object, and determining an interactive area based on the movement trajectory.

[0019] For example, in at least one embodiment of the present disclosure, an image processing method is provided, in which a corresponding interactive operation is performed based on the plurality of processed images and the interactive action, including: in response to the interactive action being the second selection action, identifying the content in the interactive area based on the plurality of processed images.

[0020] For example, in at least one embodiment of the image processing method provided in this disclosure, the segmentation model includes a multi-scale module and a backbone network module. The multi-scale module includes a first sub-module, a second sub-module, and a third sub-module. The first sub-module, the second sub-module, and the third sub-module are configured to extract features of different dimensions. Specifically, the first sub-module is configured to extract detailed features, and the second and third sub-modules are configured to extract global features. The backbone network module is configured to fuse the features extracted by the first sub-module, the second sub-module, and the third sub-module to obtain the output of the segmentation model.

[0021] For example, in at least one embodiment of the image processing method provided in this disclosure, the first submodule includes a convolutional layer and a max pooling layer with a size of 1*1; the second submodule includes a plurality of first dilated convolutional layers and a plurality of deconvolutional layers, wherein the plurality of first dilated convolutional layers are divided into a plurality of groups for group convolution; the third submodule includes a plurality of second dilated convolutional layers, wherein the plurality of third dilated convolutional layers are divided into a plurality of groups for group convolution.

[0022] At least one embodiment of this disclosure provides an image processing apparatus, comprising: an image acquisition module configured to capture interactive actions performed by an interactive action object on a recognition object using a shooting device, thereby obtaining multiple frame images; a processing module configured to perform blurring processing on the multiple frame images, thereby obtaining multiple processed images corresponding to the multiple frame images, wherein the blurring processing ensures that the interactive action object does not obstruct the displayed content of the recognition object in the processed images; a recognition module configured to recognize the interactive action of the interactive action object based on the multiple frame images; and an execution module configured to execute a corresponding interactive operation based on the multiple processed images and the interactive action.

[0023] At least one embodiment of this disclosure provides an electronic device, including a shooting device and an image processing unit, wherein the shooting device is configured to capture interactive actions performed by an interactive action object on a recognition object to obtain a video stream; the image processing unit is configured to receive the video stream, acquire the plurality of frame images from the video stream, and execute the image processing method described in any embodiment of this disclosure.

[0024] At least one embodiment of this disclosure provides an electronic device, including: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to execute the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement the image processing method according to any embodiment of this disclosure.

[0025] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the image processing method according to any embodiment of this disclosure. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure, and are not intended to limit this disclosure.

[0027] Figure 1 This is a schematic diagram showing an occluded area in a frame image.

[0028] Figure 2 A schematic flowchart illustrating an image processing method provided for at least one embodiment of this disclosure;

[0029] Figure 3 A schematic flowchart illustrating step S20 provided for at least one embodiment of this disclosure;

[0030] Figure 4 A process for acquiring reference images provided for at least one embodiment of this disclosure;

[0031] Figure 5 A schematic structural diagram of a segmentation model provided in at least one embodiment of this disclosure;

[0032] Figure 6A A schematic diagram of a processed image provided in at least one embodiment of this disclosure;

[0033] Figure 6B A schematic diagram of a gesture action provided in at least one embodiment of this disclosure;

[0034] Figure 7 A schematic flowchart illustrating step S30 provided for at least one embodiment of this disclosure;

[0035] Figure 8 A schematic diagram illustrating a first selected action provided for at least one embodiment of this disclosure;

[0036] Figure 9 A schematic block diagram of an image processing apparatus provided in at least one embodiment of the present disclosure;

[0037] Figure 10A A schematic block diagram of an electronic device provided for at least one embodiment of this disclosure;

[0038] Figure 10B A schematic block diagram of another electronic device provided for at least one embodiment of this disclosure;

[0039] Figure 11 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of this disclosure;

[0040] Figure 12 This is a schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0042] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and known components are omitted.

[0043] With the continuous development of distance learning, students can use terminal devices such as learning machines to learn. For example, the display screen of the terminal device can be a touch screen, allowing students to perform some interactive actions by touching the screen with their fingers, such as starting to play a video, stopping the video, or selecting an area of ​​the screen for content recognition.

[0044] However, in the actual learning process, students still need to use physical learning materials for assistance, such as test papers, workbooks, and textbooks. Since interactive information cannot be provided through touchscreens or other I / O interfaces, learning machines and other terminal devices cannot interactively process these physical learning materials, which reduces the intelligence of the terminal devices.

[0045] Furthermore, during the interaction process, the user's hand or other auxiliary parts, such as a pen, may obstruct the physical learning materials. As a result, the terminal device cannot extract all the content of the physical materials during the interaction process, which will bring certain obstacles to the recognition of interactive actions, action areas, and content in the action areas of the physical materials.

[0046] For example, in one use case, a user circles the location of an incorrect question on a test paper with their finger. The terminal device needs to recognize the user's action as circling the question, identify the interactive area selected by the finger, and recognize the question content within that interactive area to provide the correct answer to the question. In this case, such as... Figure 1As shown, because the user's hand obstructs the content of the test paper during the process of circling the question, for example, the hand obscures the content to be identified in the test paper during the movement of the hand, this will directly lead to the inability to accurately locate the interactive area selected by the user when circling the question, and thus the inability to obtain the actual question area selected by the user, as well as the accurate question content, and therefore the inability to provide the correct answer to the question.

[0047] In other words, when a user interacts with physical objects using interactive objects such as hands or pens, if the interactive object obscures the content of the physical object, the interaction area when the user performs the interactive action cannot be accurately obtained, and the content of the interaction area cannot be accurately identified.

[0048] This disclosure provides at least one embodiment of an image processing method, an image processing apparatus, an electronic device, and a non-transitory computer-readable storage medium. The image processing method includes: capturing interactive actions performed by an interactive action object on a recognition object using an imaging device to obtain multiple frame images; recognizing the interactive actions of the interactive action object based on the multiple frame images; blurring the multiple frame images to obtain multiple processed images corresponding to each frame image, wherein the blurring process ensures that the interactive action object does not obstruct the displayed content of the recognition object in each processed image; and performing a corresponding interactive operation based on the multiple processed images and the interactive actions.

[0049] In at least one embodiment, the image processing method can identify the interactive actions performed by the interactive action object on the recognition object. For example, it can accurately identify the interactive area selected on the recognition object when the interactive action object performs the interactive action, thereby accurately executing the corresponding interactive operation, such as identifying the content in the interactive area. This provides a kind of air-to-air interactive operation for terminal devices such as learning machines, rather than being limited to completing the interaction only through the I / O interface, thus improving intelligence.

[0050] In addition, multiple frames of images captured are blurred so that the interactive action object does not obstruct the displayed content of the recognition object in each processed image. That is, in the processed image, the displayed content of the recognition object is fully visible and is not affected by the interactive action object. When the interactive action is a selection action, the interactive area is accurately located and the content in the interactive area is recognized, thus improving the accuracy of recognition.

[0051] The image processing method provided in this disclosure can be applied to the image processing apparatus provided in this disclosure, which can be configured on an electronic device. The electronic device can be a personal computer, server, mobile terminal, etc. For example, the mobile terminal can be a mobile phone, tablet computer, or other hardware device; for instance, the hardware device can be a learning machine or other terminal device.

[0052] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0053] Figure 2 This is a schematic flowchart illustrating an image processing method provided in at least one embodiment of the present disclosure.

[0054] like Figure 2 As shown, the image processing method provided in at least one embodiment of this disclosure includes steps S10 to S40. It should be noted that steps S20 and S30 in this disclosure can be processed in parallel, without any restriction on their order.

[0055] In step S10, the camera is used to capture the interactive actions performed by the interactive action object on the identified object, resulting in multiple frame images.

[0056] For example, the shooting device can be a front-facing camera or a rear-facing camera configured on the terminal device.

[0057] For example, when the shooting device is a front-facing camera, in order for the front-facing camera to be able to capture the recognition object (such as test papers, exercise books, or other physical materials) placed on the plane on which the terminal device is placed, a reflective device, such as a mirror, can be set on the front-facing camera so that the shooting device can capture the plane on which the recognition object is located.

[0058] For example, a video stream is recorded using a camera device when an interactive object performs an interactive action on a recognized object. This video stream includes multiple frames, which are then used as multiple frame images for subsequent processing.

[0059] For example, multiple keyframes can be extracted from multiple frames of a video stream as multiple frame images for subsequent processing. The method for determining multiple keyframes can be determined according to actual needs, and this disclosure does not impose any restrictions on this. In this case, the multiple frame images may not be consecutive frames in the video stream.

[0060] In other words, multiple frames can be sequential in time or discontinuous in time, and this disclosure does not impose any limitation on this.

[0061] For example, the interactive action object can be a gesture action object, such as the user's hand, or it can be other auxiliary objects, such as a pen, etc. This disclosure does not impose any specific restrictions on it.

[0062] For example, the object to be identified can be a physical document with displayed content, such as an exam paper, workbook, or book. Alternatively, the object can be the display screen of another electronic device, where content is displayed. The displayed content can include any visible content such as text, images, formulas, or tables within the object to be identified; this disclosure does not impose specific limitations on the displayed content.

[0063] In step S20, multiple frame images are blurred to obtain multiple processed images corresponding to the multiple frame images.

[0064] For example, blurring ensures that in each processed image, the interactive action object does not obscure the displayed content in the recognized object.

[0065] For example, "not constituting occlusion" here could mean that the interactive action object still exists in the processed image, but its transparency is high, and the displayed content in the recognized object is still fully visible. Visually, it is more like adding an image of a highly transparent interactive action object to the recognized object, and the displayed content (such as text, images, formulas, etc.) in the original recognized object at the location of the interactive action object is still fully displayed.

[0066] Therefore, since the interactive action object no longer obstructs the displayed content in the recognition object, the interactive area selected by the interactive action object and the content to be recognized in the interactive area can be accurately identified.

[0067] Figure 3 This is a schematic flowchart illustrating step S20 provided for at least one embodiment of the present disclosure.

[0068] like Figure 3 As shown, step S20 may include at least steps S201-S204.

[0069] In step S201, the reference image corresponding to the object to be identified is determined.

[0070] For example, the reference image includes at least a portion of the display content corresponding to the identified object. This at least portion of the display content includes the display content within an occluded area of ​​the frame image, which is at least partly composed of the interactive action object. In other words, the reference image needs to contain the display content occluded by the interactive action object.

[0071] For example, in some examples, step S201 may include: taking an image of the entire display content including the identified object as a reference image.

[0072] For example, before the user performs an interactive action on the object, an image of the entire display content including the object to be identified can be captured by a camera device. For instance, if the user has not placed any interactive action object on top of the object to be identified, the captured image can be used as a reference image.

[0073] For example, in other examples, step S201 may include: sequentially performing background merging processing on multiple frame images to obtain a reference image.

[0074] For example, performing background merging processing on multiple frame images sequentially to obtain a reference image may include: for the current frame image undergoing background merging processing, using a segmentation model to determine the occlusion region in the current frame image that is at least partly composed of the interactive action object; marking the pixels in the occlusion region of the current frame image to obtain a second intermediate image corresponding to the frame image; obtaining a historical background merging image, wherein, in response to the current frame image being the first image to undergo background merging processing, the second intermediate image corresponding to the current frame image is used as the historical background merging image; performing matching processing on the historical background merging image and the second intermediate image, in response to the matching processing result indicating that the identified object has not moved, performing background fusion processing on the second intermediate image and the historical background merging image to obtain a fused image, and in response to the proportion of pixels marked as occlusion positions in the fused image being less than a proportion threshold, determining the fused image as the reference image, otherwise updating the historical background merging image to the fused image, and continuing to perform background merging processing on the next frame image; in response to the matching processing result indicating that the identified object has moved, updating the historical background merging image to the second intermediate image, and continuing to perform background merging processing on the next frame image.

[0075] For example, matching a historical background merged image and a second intermediate image can include: determining the first histogram vector corresponding to the historical background merged image and the second histogram vector corresponding to the second intermediate image; comparing the similarity between the first histogram vector and the second histogram vector, and determining the result of the matching process based on the similarity.

[0076] For example, comparing the similarity between a first histogram vector and a second histogram vector, and determining the matching result based on the similarity, may include: calculating the correlation coefficient between the first histogram vector and the second histogram vector; calculating the Euclidean distance between the first histogram vector and the second histogram vector; calculating the weighted average of the correlation coefficient and the Euclidean distance as the similarity; and determining that the matching result indicates that the identified object has not moved if the similarity is greater than a similarity threshold, and determining that the matching result indicates that the identified object has moved if the similarity is less than a similarity threshold.

[0077] For example, the similarity threshold can be 0.69. The similarity threshold can be set according to the actual situation, and this disclosure does not impose any restrictions on it.

[0078] For example, performing background fusion on a second intermediate image and a historical background merged image to obtain a fused image may include: determining one or more pixels in the historical background merged image that are marked as occlusion positions; determining pixels in the second intermediate image that correspond to one or more pixels as replacement pixels; determining at least one pixel in the replacement pixels that is not marked as an occlusion position; and replacing the value of the pixel in the one or more pixels that corresponds to at least one pixel with the value of at least one pixel to obtain the fused image.

[0079] The process of background merging is explained in detail below with reference to the attached diagram.

[0080] Figure 4 The process for acquiring reference images provided for at least one embodiment of this disclosure.

[0081] like Figure 4 As shown, a video stream is obtained by continuously capturing interactive actions performed by an interactive object on a recognition object using a shooting device. The multiple frames included in the video stream are designated as multiple frame images. For example, according to the chronological order of the shooting time, the multiple frame images are sequentially designated as frame image 1, frame image 2, ..., frame image i, frame image i+1, ..., frame image n, where n is a positive integer and represents the total number of frame images, and i is a positive integer less than n and greater than 1.

[0082] For example, for frame image 1, the segmentation model is first used to determine the occlusion region formed by the interactive action object in frame image 1. For example, when the interactive action object is a gesture action object, the segmentation model can be a hand recognition model, such as a skin detection model, etc. The segmentation model is used to identify the hand region from the frame image and uses the identified hand region as the occlusion region. Of course, when the interactive action object is other auxiliary objects, such as a pen, the segmentation model can be pre-trained so that the segmentation model can segment the image of the auxiliary object, identify the auxiliary object region from the frame image, and use it as the occlusion region.

[0083] Figure 5 A schematic structural diagram of a segmentation model provided for at least one embodiment of this disclosure.

[0084] like Figure 5 As shown, the segmentation model 100 includes a multi-scale module 101 and a backbone network module 102.

[0085] The multi-scale module 101 includes a first sub-module 1011, a second sub-module 1012, and a third sub-module 1013, which are configured to extract features of different dimensions.

[0086] For example, the first submodule 1011 is configured to extract detailed features, while the second submodule 1012 and the third submodule 1013 are configured to extract global features.

[0087] For example, such as Figure 5 As shown, the first submodule 1011 includes a convolutional layer and a max pooling layer with a size of 1*1.

[0088] The second submodule 1012 includes multiple first-dilated convolutional layers and multiple deconvolutional layers, such as two 3*3 dilated convolutional layers and two 3*3 deconvolutional layers. Here, the multiple first-dilated convolutional layers are divided into multiple groups for grouped convolution, for example, divided into 10 groups for grouped convolution, to further reduce the computational cost when using dilated convolution. The specific process of implementing grouping and the corresponding grouped convolution can be referred to the typical processing procedure for grouped convolution, and will not be elaborated here.

[0089] The third submodule includes multiple second-dilated convolutional layers, such as four 3x3 dilated convolutional layers. Here, these multiple second-dilated convolutional layers are divided into multiple groups for grouped convolution, for example, divided into 10 groups for grouped convolution, to further reduce the computational cost.

[0090] For example, the backbone network module 102 is configured to fuse features extracted from the first, second, and third sub-modules to obtain the output of the segmentation model. For example, the backbone network module includes two 3*3 deconvolutional layers, and is a lightweight feature extraction module, which significantly reduces the computational resources it occupies.

[0091] In traditional segmentation models, multi-scale modules are typically configured to convert input images into images of different sizes and input them into the backbone network module. Ultimately, the backbone network still extracts features from the images of different sizes and fuses them. In this approach, feature extraction still relies heavily on the backbone network module, making it difficult to compress the model during the actual implementation of the algorithm, resulting in high overall computational costs.

[0092] However, the segmentation model provided in this disclosure utilizes a first sub-module, a second sub-module, and a third sub-module capable of extracting features of different dimensions to achieve multi-scale processing, thereby shifting the dependency of feature extraction to the preceding multi-scale modules. For example, in this disclosure, a 1*1 convolutional layer in the first sub-module is used to extract low-dimensional features, which have low dimensionality, high resolution, and a small receptive field, enabling the extraction of detailed features containing local information; a 3*3 dilated convolutional layer in the second and third sub-modules is used to extract high-dimensional features, which have high dimensionality, high abstraction, and a large receptive field, enabling the extraction of global features containing global information and supplementing multi-scale contextual information; in addition, a deconvolutional layer is set to expand the area of ​​the feature map to extract more details.

[0093] Furthermore, this disclosure also incorporates convolution groups to further reduce the computational cost when using dilated convolutions. Since the multi-scale module setting allows for the use of a lighter backbone network module, the segmentation model provided by this disclosure is more conducive to mobile deployment, reducing computational resource consumption and improving computational efficiency.

[0094] Of course, this disclosure is not limited to this, and other forms of segmentation models can also be used to extract occluded regions. This disclosure does not impose any specific restrictions on this.

[0095] Next, the pixels in the occluded area of ​​frame image 1 are marked. For example, the pixel values ​​of the pixels in the occluded area are marked as preset values, such as 0, to obtain the second intermediate image 1 corresponding to frame image 1.

[0096] Since frame image 1 is the first image to undergo background merging, the second intermediate image 1 is used as the historical background merging image. Then, background merging is performed on frame image 2.

[0097] For frame image 2, the occlusion region in frame image 2 is first determined using a segmentation model. The specific process is as described above and will not be repeated here.

[0098] Next, the pixels in the occluded area of ​​frame image 2 are marked. For example, all the pixel values ​​of the pixels in the occluded area of ​​frame image 2 are marked with a preset value (e.g., 0) to obtain the second intermediate image 2 corresponding to frame image 2.

[0099] Since the object to be identified may move during the shooting process, if it moves, continuing to use the historical merging results obtained in the previous stage for subsequent background fusion may result in a large error. Therefore, it is necessary to perform matching processing on the historical background merging image and the second intermediate image to determine whether the object to be identified has moved based on the matching processing results.

[0100] For example, when performing matching processing on the historical background merged image (i.e., the second intermediate image 1) and the second intermediate image 2, the histogram vectors in the historical background merged image and the second intermediate image 2 are used as the matching criteria.

[0101] Specifically, first, determine the first histogram vector corresponding to the merged historical background image and the second histogram vector corresponding to the second intermediate image 2. For example, the histogram vector represents a 1*M dimensional vector generated after histogram statistics, where M is a positive integer and represents the color depth of the pixel, for example, M is 256 (2 8 In a histogram vector, each element value represents the number of pixels corresponding to that pixel value. For example, when calculating the histogram vector between the merged historical background image and the second intermediate image 2, the pixel values ​​of the occluded areas are set to a preset value, such as 0. Therefore, the dimension corresponding to the preset value can be ignored in the histogram vector, thus converting the histogram vector into a 1*(M-1) dimension.

[0102] Then, the matching result is determined based on the similarity between the first and second histogram vectors.

[0103] In this disclosure, two algorithms, relevance comparison and Euclidean distance, are used to simultaneously calculate the similarity between the first and second histogram vectors to enhance the accuracy of the similarity.

[0104] For example, the correlation coefficient between the first and second histogram vectors is calculated. The correlation coefficient can be calculated using the Pearson correlation coefficient formula, etc., and this disclosure does not impose specific limitations on this. The Euclidean distance between the first and second histogram vectors is calculated. The weighted average of the correlation coefficient and the Euclidean distance is calculated as the similarity. If the similarity is greater than the similarity threshold, it indicates that the first and second histogram vectors are highly similar, and the result of the matching process indicates that the identified object has not moved. If the similarity is less than the similarity threshold, it indicates that the first and second histogram vectors are less similar, and the result of the matching process indicates that the identified object has moved.

[0105] Subsequently, if the matching process indicates that the object being identified has not moved, the second intermediate image 2 and the historical background merged image are merged to obtain a fused image.

[0106] Specifically, first, P pixels marked as occlusion positions in the historical background merged image are determined, where P is a positive integer. For example, the pixel values ​​of these P pixels are the aforementioned preset values, such as 0, indicating that the P pixels are occluded by the interactive object and the pixel values ​​to be restored are unknown. Then, P replacement pixels in the second intermediate image 2 corresponding to these P pixels are determined. Here, "corresponding" means that the pixels located at the same position in the historical background merged image and the second intermediate image 2 are called "corresponding". From the P replacement pixels, Q replacement pixels that are not marked with the preset value are selected, where Q is a positive integer and less than or equal to P. The pixel values ​​of the Q pixels in the historical background merged image that correspond to the Q replacement pixels are replaced with the pixel values ​​of the Q replacement pixels in the second intermediate image 2, thereby obtaining the merged image.

[0107] In one embodiment, during background fusion, pixels in the historical background merged image that are occluded by the interactive object and whose values ​​are still unknown to be recovered can be marked as 0, while other pixels are marked as 1. In the second intermediate image 2, pixels marked as 0 in the historical background merged image but not occluded by the interactive object in the second intermediate image 2 are searched and replaced with the corresponding pixels in the historical background merged image. That is, the pixel value of the pixel marked as 0 in the historical background merged image is modified to the pixel value of the corresponding pixel in the second intermediate image 2. Simultaneously, this pixel is marked as 1 in the historical background merged image, indicating that its pixel value has been recovered. Thus, the fused image is obtained.

[0108] Subsequently, if the proportion of pixels marked as occluded positions in the fused image is less than the proportion threshold, the fused image is determined as the reference image; otherwise, the historical background merging image is updated to the fused image, and the background merging process continues for the next frame image (i.e., frame image 3). That is, the fused image (i.e., the fused image obtained by performing background merging processing on frame image 2) is used as the historical background merging image for the next frame image (i.e., frame image 3) to perform background merging processing. The specific process will not be described in detail.

[0109] For example, the ratio threshold can be set by the user according to the actual situation; for example, the ratio threshold can be 99%.

[0110] It should be noted that, in the embodiments of this disclosure, "percentage" refers to the ratio of the number of pixels marked as occluded positions in the fused image to the total number of pixels in the fused image.

[0111] For example, if the matching process indicates that the object being identified has moved, the historical background merged image is updated to the second intermediate image 2, which is the merged result before being cleared, and the background merging process described above is continued for the next frame image. The specific process will not be described in detail here.

[0112] Then, without obtaining a reference image, the above process continues until a reference image is output, which will not be described in detail here.

[0113] If a reference image is still not obtained after performing the above processing on multiple video frames, the background may be changing too frequently. For example, you can continue to acquire new video frames and perform the above processing until a reference image is obtained.

[0114] It should be noted that if it is stipulated in advance that the object to be identified will not move, then during the background fusion process, matching processing can be omitted. After obtaining the second intermediate image and the historical background merged image, the second intermediate image and the historical background merged image can be directly fused to obtain the fused image. The subsequent process is the same as the above content, and will not be repeated here.

[0115] After the aforementioned background merging process, since the interactive action object may move in multiple frame images, the occlusion area caused by the movement of the interactive action object may differ in different frame images. Therefore, background information extracted from different frame images can be used to stitch together a reference image, ensuring that the reference image includes some or all of the display content of the identified object. Thus, even if the interactive action object occludes the identified object during interaction, it does not affect the acquisition of the final identified content, improving recognition accuracy and enabling precise location of the interactive action object's interaction area and the content to be identified within that area, thereby enhancing product intelligence. Furthermore, this disclosure is not limited to whether the identified object moves; even if the identified object moves during interaction, a reference image containing accurate display content can still be obtained, improving the flexibility of the method execution and expanding its applicability to various scenarios.

[0116] After obtaining the reference image, perform the following steps on each frame image: Figure 3 The steps S202-S204 shown are used to obtain the processed image corresponding to each frame image.

[0117] In step S202, the occlusion region in the frame image, which is at least part of the interactive action object, is determined using a segmentation model.

[0118] For example, at least part of the displayed content includes the content displayed in the occluded area. That is, the reference image must at least include the text, images, formulas, and other displayed content in the identified object that are occluded by the interactive object.

[0119] For example, the process of determining the occlusion area using the segmentation model can be referred to the relevant description in step S201, which will not be repeated here.

[0120] In step S203, the reference image and the occluded area are merged to obtain the first intermediate image corresponding to the frame image.

[0121] For example, step S203 may include: adding the display content corresponding to the occluded area in the frame image to the reference image to obtain the first intermediate image corresponding to the frame image.

[0122] For example, if the interactive action object is a gesture action object, the region where the gesture action object is located, i.e., the occluded region, is segmented from the frame image in step S202. The display content corresponding to the occluded region is then the image of the gesture action object. The image of the gesture action object is added to the corresponding position in the reference image, for example, by adding the image of the gesture action object to the corresponding position in the reference image as an overlay layer, to obtain the first intermediate image.

[0123] In step S204, the occluded area in the first intermediate image is blurred to obtain the processed image corresponding to the frame image.

[0124] For example, step S204 may include: increasing the transparency of the occluded area in the first intermediate image to obtain the processed image corresponding to the frame image.

[0125] For example, the occluded area in the first intermediate image is the display content corresponding to the occluded area in the frame image added in step S203, such as the image of the gesture object.

[0126] For example, taking the interactive action object as the gesture action object, for the first intermediate image, the transparency of the image of the gesture action object is increased to make it semi-transparent, so that it will not obscure the display content at the corresponding position in the reference image. The display content at the corresponding position in the reference image is still fully visible. Thus, the processed image corresponding to the frame image is obtained.

[0127] Figure 6A This is a schematic diagram of a processed image provided in at least one embodiment of this disclosure. Figure 6A As shown, the interactive action object is a gesture action object, and the recognition object is a page in a book. The gesture action object moves on the recognition object to select the area of ​​the question to be recognized. For a certain frame image, in its corresponding processed image, the image of the gesture action object is in a semi-transparent state. The display content in the reference image at the position of the gesture action object (that is, the display content in the original recognition object, such as text, pictures, formulas, etc.) is still fully visible and will not be obscured by the gesture action object.

[0128] Of course, it should be noted that steps S202-S204 can be performed on some or all of the multiple frame images, that is, the processed images corresponding to some or all of the frame images can be obtained for subsequent operations. In practice, you can choose as needed, and this disclosure does not impose any specific restrictions on this.

[0129] In step S30, the interactive actions of the interactive action object are identified based on multiple frame images.

[0130] For example, the interactive action of an interactive action object includes at least the interactive area selected on the identified object when the interactive action object performs the interactive action.

[0131] For example, multiple interactive actions can be predefined, and when an interactive action object is detected to perform an interactive action, the corresponding interactive operation is triggered.

[0132] For example, Figure 6B This is a schematic diagram of a gesture action provided in at least one embodiment of the present disclosure.

[0133] Figure 6B In the diagram, (a) represents the select gesture, (b) represents the confirm gesture, (c) represents the pause gesture, and (d) represents the stop gesture.

[0134] For example, different interactive actions can be defined by combining the various gestures mentioned above.

[0135] For example, in one scenario, when a user wants to select an area of ​​incorrect questions on a test paper to retrieve the correct answers, the user utilizes... Figure 6B In (a), the selection gesture is used to select the area of ​​incorrect questions. After selection, use the following method: Figure 6B The stop gesture in (d) is shown. At this point, by recognizing the changes in gesture movements across multiple frames, it is confirmed that the gesture action object is performing the selection action for the corresponding question.

[0136] The specific process for recognizing interactive actions can utilize any feasible method, such as detecting gestures in each frame of an image and determining the user's interactive action based on changes in gestures across multiple frames. This disclosure does not impose specific limitations on the specific method for recognizing interactive actions.

[0137] certainly, Figure 6B The gesture definitions shown are only illustrative descriptions. Other or more gesture definitions may be used as needed, or other different interactive actions may be defined. This disclosure does not impose any specific restrictions on this.

[0138] Figure 7 This is a schematic flowchart illustrating step S30 provided for at least one embodiment of the present disclosure.

[0139] like Figure 7 As shown, step S30 includes at least step S301 and step S302.

[0140] In step S301: For each frame image, determine the position of the key points of the interactive action object in the frame image.

[0141] For example, the key points of the interactive action object can be set as needed. For example, when the interactive action object is a gesture action object, the key point of the interactive action object can be the fingertip of the target finger, such as the fingertip of the index finger; for example, when the interactive action object is a pen, the key point of the interactive action object can be the pen tip. This disclosure does not impose any specific restrictions on this.

[0142] For example, when the interactive action object is a gesture action object, the key points may be unstable, that is, it cannot be guaranteed that the same relative position can be extracted as the key point every time. This also leads to the instability of the determined interactive area, which affects subsequent content recognition and other processes.

[0143] For example, step S301 may include: using a keypoint detection model to identify target keypoints in a gesture action object; extracting a detection region containing the target keypoints in a frame image based on the target keypoints; extracting at least one corner feature position within the detection region; performing Gaussian convolution calculation at each corner feature position to obtain a corner feature map corresponding to the frame image; performing a two-dimensional convolution calculation between the corner feature map and the corner feature map corresponding to a reference frame image to obtain a calculation result, wherein the reference frame image is a frame image that is located before and adjacent to the frame image in the shooting order among multiple frame images; determining the position corresponding to the maximum value in the calculation result; and correcting the position of the target keypoint based on the position corresponding to the maximum value to obtain the position of the keypoints of the gesture action object.

[0144] Specifically, the method first uses a keypoint detection model to identify target keypoints in the gesture action object. The keypoint detection model can be any feasible finger keypoint detection model, and this disclosure does not impose any specific restrictions on it. For example, the target keypoint can be the keypoint at the tip of the index finger.

[0145] Next, the detection region including the target key point is extracted from the frame image. For example, the region with a size of 20*20 centered on the target key point in the frame image is extracted as the detection region.

[0146] Next, at least one corner feature position is extracted within the detection area. The corner feature position is the extreme point of certain attributes in the detection area. For example, any feasible corner detection algorithm can be used to extract the corner feature position, and this disclosure does not impose any specific limitations on it.

[0147] Next, Gaussian convolution is performed at each corner feature location to obtain the corner feature map corresponding to the frame image. For example, performing a 5*5 Gaussian convolution at each corner feature location yields the corner feature map corresponding to the detection region in that frame image. After Gaussian convolution, the features at the corner feature locations are further highlighted, which is beneficial for extracting relatively accurate offsets in subsequent steps.

[0148] The corner feature map is convolved with the corner feature map corresponding to the reference frame image in two dimensions to obtain the calculation result. For example, if the frame image being processed is frame image i+1, then its reference frame image is frame image i.

[0149] The location corresponding to the maximum value in the calculation result is determined, and the position of the target key point is corrected based on the location of the maximum value. For example, the location corresponding to the maximum value is the coordinate of the maximum value in the two-dimensional coordinate system formed by the two-dimensional convolution result. It represents the positional offset of the target key point relative to the reference frame image, that is, the offset in the x-axis direction and y-axis direction.

[0150] Through two-dimensional convolution calculation, the position change information of the target key points in two adjacent frame images is extracted. The most matching position in the reference frame image and the currently processed frame image is determined to correct the target key points, thereby stabilizing the position of the key points of the gesture object and avoiding the key points from jumping in different frame images. This helps to select the accurate interaction area and improve the accuracy of recognition.

[0151] Step S302: Identify the interactive action based on the position of the key points of the interactive action object in multiple frame images.

[0152] Then, in step S40, the corresponding interactive operation is executed based on the multiple processed images and interactive actions.

[0153] This disclosure provides two scenarios for interactive action recognition. The recognition of the two interactive actions and the execution of subsequent interactive operations are described below in conjunction with steps S302 and S40, respectively.

[0154] For example, in some examples, the interactive action object includes a first hand and a second hand, and the key points of the interactive action object include a first key point corresponding to the first hand and a second key point corresponding to the second hand. In this case, step S302 may include: in response to the duration of the position changes of both the first key point and the second key point being less than a position threshold being greater than a time threshold: determining the interactive action as a first selection action, and determining the interactive area based on the positions of the first key point and the second key point.

[0155] For example, determining the interaction area based on the positions of the first key point and the second key point can include: determining the bounding box based on the positions of the first key point and the second key point, and using the bounding box as the interaction area.

[0156] For example, the circumscribed region box can be a circular region box with the first key point and the second key point as the two ends of the diameter, or a rectangular box with the first key point and the second key point as the diagonal points, or an elliptical region box with the first key point and the second key point as the foci of the ellipse, etc. This disclosure does not impose specific restrictions on the shape or acquisition method of the circumscribed region box.

[0157] For example, step S40 may include: in response to an interactive action being a first selection action, obtaining a text description of the content in the interactive area based on multiple processed images.

[0158] For example, any feasible intelligent image recognition algorithm (such as image caption) can be used to recognize multiple processed images and obtain text descriptions of the content in the interactive area, such as text descriptions of images in the interactive area.

[0159] Figure 8 This is a schematic diagram of a first selected action provided for at least one embodiment of the present disclosure.

[0160] like Figure 8 As shown, the interactive action object includes two hands, namely the first hand and the second hand. Through the above step S301, the first key point corresponding to the first hand and the second key point corresponding to the second hand are obtained.

[0161] For example, if the positions of the first and second keypoints remain fixed for more than n seconds (a time threshold) across multiple consecutive frames, or if the positions of the first and second keypoints change very little across multiple consecutive frames (e.g., only minor movements occur, with position changes less than the position threshold), and the duration of this state exceeds n seconds, then the interaction action is determined as the first selected action. Here, n is a positive number and can be set according to actual needs.

[0162] In this disclosure, the time threshold refers to the temporal relationship between multiple frame images, and the position threshold, for example, refers to the absolute positional relationship between the first keypoint and the second keypoint between multiple frame images. For example, if, relative to the position of the first keypoint in the initial frame image (e.g., frame image 1), the position of the first keypoint is within a range of m*n centered on the position of the first keypoint in frame image 1 (e.g., m is 3 pixels and n is 3 pixels) for u consecutive frame images, and, relative to the position of the second keypoint in frame image 1, the position of the second keypoint is within a range of m*n centered on the position of the second keypoint in frame image 1 for u consecutive frame images, then the interaction action is determined to be the first selection action. Here, the product of u and the frame length is greater than or equal to n, and u is a positive integer.

[0163] And, as Figure 8 As shown, the rectangular frame with the first key point and the second key point as diagonal points is used as the interactive area. In step S40, the image in the interactive area is identified by the intelligent image recognition algorithm to obtain the text description of the image ("Acity is one of the visited tourist attractions").

[0164] Therefore, this disclosure provides a scenario interaction method capable of describing objects. After the gesture object is fixed for n seconds or more, the interaction area is obtained from the key points in the gesture object, and the corresponding text description is obtained by recognizing the content in the interaction area.

[0165] Of course, the content in the interactive area can also be identified based on the interactive area obtained in the above embodiments, and this disclosure does not impose any specific limitations on this.

[0166] For example, in other examples, step S302 may include: in response to the key point of the interactive action object moving on the identified object and detecting a stop action: determining the interactive action as a second selected action, and determining the movement trajectory of the key point of the interactive action object between the stop actions based on the position of the key point of the interactive action object, and determining the interactive area based on the movement trajectory.

[0167] For example, the stopping action can be Figure 6B The stop gesture shown in (d) can be any other defined state or action, and this disclosure does not impose any specific limitations on it.

[0168] For example, the area enclosed by the key points of the interactive action object, such as the tip of the user's index finger, when moving on the object can be used as the interaction area.

[0169] For example, step S40 may include: in response to an interactive action being a second selection action, identifying content in the interactive area based on multiple processed images.

[0170] Since the interactive objects no longer obstruct the displayed content in the processed image, the content in the interactive area can be accurately identified, such as text, images, tables, formulas, etc.

[0171] In some embodiments, such as when the second selection action is the question-circling action as described above, the correct answer to the question can be retrieved from the database based on the content in the identified interactive area and displayed on the screen.

[0172] In the image processing method provided in at least one embodiment of this disclosure, background information can be extracted from each of multiple frame images and stitched together to form a reference image to obtain all the display content of the object to be identified. A processed image is obtained by combining the reference image, such that the interactive object is semi-transparent in the processed image without obstructing the display content of the object to be identified. Multiple frame images are used to determine the user's interactive action on the object to be identified, thereby enabling air interaction. Furthermore, when the user's interactive action is a selection action, the processed image without obstruction can accurately obtain the interaction area and its content, improving recognition accuracy and processing efficiency.

[0173] At least one embodiment of this disclosure also provides an image processing apparatus. Figure 9 This is a schematic block diagram of an image processing apparatus provided for at least one embodiment of the present disclosure.

[0174] like Figure 9 As shown, the image processing apparatus 200 may include an image acquisition module 201, a processing module 202, a recognition module 203, and an execution module 204. These components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). For example, these modules can be implemented by hardware (e.g., circuit) modules, software modules, or any combination of both, as is the case in the following embodiments, and will not be described again. For example, these units can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor (TPU), a field-programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, along with corresponding computer instructions. It should be noted that the components and structure of the image processing apparatus 200 shown in FIG. 9 are exemplary and not limiting; the image processing apparatus 200 may also have other components and structures as needed.

[0175] For example, the image acquisition module 201 is configured to use a shooting device to capture the interactive actions performed by the interactive action object on the recognition object, thereby obtaining multiple frame images.

[0176] For example, processing module 202 is configured to perform blurring processing on multiple frame images to obtain multiple processed images corresponding to the multiple frame images. For example, the blurring processing ensures that the interactive action object does not obscure the displayed content of the recognition object in the processed image.

[0177] For example, the recognition module 203 is configured to recognize the interactive actions of an interactive action object based on multiple frame images.

[0178] For example, execution module 204 is configured to perform corresponding interactive operations based on multiple processed images and interactive actions.

[0179] For example, the image acquisition module 201, processing module 202, recognition module 203, and execution module 204 may include code and programs stored in memory; the processor may execute the code and programs to implement some or all of the functions of the image acquisition module 201, processing module 202, recognition module 203, and execution module 204 as described above. For example, the image acquisition module 201, processing module 202, recognition module 203, and execution module 204 may be dedicated hardware devices used to implement some or all of the functions of the image acquisition module 201, processing module 202, recognition module 203, and execution module 204 as described above. For example, the image acquisition module 201, processing module 202, recognition module 203, and execution module 204 may be a circuit board or a combination of multiple circuit boards used to implement the functions described above. In the embodiments of this application, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) processor-executable firmware stored in memory.

[0180] It should be noted that the image acquisition module 201 can be used to implement Figure 2 The processing module 202 can be used to implement step S10 shown. Figure 2 The identification module 203 can be used to implement step S20 as shown. Figure 2 The execution module 204 can be used to implement step S30 as shown. Figure 2The step S40 is shown. Therefore, for a detailed description of the functions that the image acquisition module 201, processing module 202, recognition module 203, and execution module 204 can achieve, please refer to the relevant descriptions of steps S10 to S40 in the embodiments of the above image processing method; repeated details will not be repeated here. Furthermore, the image processing apparatus 200 can achieve similar technical effects to the aforementioned image processing method, which will not be described further here.

[0181] It should be noted that, in the embodiments of this disclosure, the image processing apparatus 200 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited; it can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0182] This disclosure also provides an electronic device in some embodiments. Figure 10A This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0183] For example, such as Figure 10A As shown, the electronic device 300 includes a shooting device 301 and an image processing unit 302.

[0184] For example, the shooting device 301 is configured to capture the interactive actions performed by the interactive action object on the recognized object, thereby obtaining a video stream.

[0185] For example, the image processing unit 302 is configured to receive a video stream, acquire multiple frame images from the video stream, and execute the image processing method as described in any of the above embodiments.

[0186] For example, the shooting device 301 can be a rear camera of an electronic device, or a front camera and reflective device of an electronic device. For a related introduction to the shooting device 301, please refer to the foregoing content, which will not be repeated here.

[0187] For example, the image processing unit 302 can be implemented as a central processing unit, a dedicated processing chip, a digital signal processor, etc., and this disclosure does not impose any specific limitations on it.

[0188] For example, electronic device 300 can be a terminal device such as a learning machine, and can also provide a display unit (such as a touch screen). For example, the display unit can provide a corresponding human-computer interaction interface for displaying the response of interactive operation, interactive action prompt information, etc. This disclosure does not impose specific limitations in this regard.

[0189] For example, a detailed description of the process by which the electronic device 300 performs the image processing method can be found in the relevant description in the embodiments of the above-described image processing method, and repeated descriptions will not be repeated here.

[0190] Some embodiments of this disclosure also provide another electronic device. Figure 10B This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0191] For example, such as Figure 10B As shown, the electronic device 400 includes a processor 401 and a memory 402. It should be noted that... Figure 10B The components of the electronic device 400 shown are merely exemplary and not limiting. The electronic device 400 may have other components as needed for the actual application.

[0192] For example, processor 401 and memory 402 can communicate with each other directly or indirectly.

[0193] For example, processor 401 and memory 402 can communicate via a network. The network can include wireless networks, wired networks, and / or any combination of wireless and wired networks. Processor 401 and memory 402 can also communicate with each other via a system bus, and this disclosure is not limiting in this regard.

[0194] For example, in some embodiments, memory 402 is used to store computer-readable instructions non-transitory. When processor 401 executes the computer-readable instructions, the computer-readable instructions are executed by processor 401 to implement the image processing method according to any of the above embodiments. For specific implementations and related explanations of the various steps of this image processing method, please refer to the embodiments of the image processing method described above; repeated details will not be elaborated here.

[0195] For example, processor 401 and memory 402 can be located on the server side (or in the cloud).

[0196] For example, processor 401 can control other components in electronic device 400 to perform desired functions. Processor 401 can be a central processing unit (CPU), graphics processing unit (GPU), network processor (NP), etc.; it can also be a digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) can be based on x86 or ARM architectures, etc.

[0197] For example, memory 402 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and processor 401 may execute the computer-readable instructions to implement various functions of electronic device 400. Various application programs and various data may also be stored in the storage medium.

[0198] For example, in some embodiments, the electronic device 400 can be a mobile phone, tablet computer, electronic paper, television, monitor, laptop computer, digital photo frame, navigator, wearable electronic device, smart home device, etc.

[0199] For example, electronic device 400 may include a display panel, which can be used for image segmentation, etc. For example, the display panel can be a rectangular panel, a circular panel, an elliptical panel, or a polygonal panel. Furthermore, the display panel can be not only a flat panel, but also a curved panel, or even a spherical panel.

[0200] For example, electronic device 400 can have touch functionality, that is, electronic device 400 can be a touch device.

[0201] For example, a detailed description of the process by which the electronic device 400 performs the image processing method can be found in the relevant description in the embodiments of the above-described image processing method, and repeated descriptions will not be repeated here.

[0202] Figure 11 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. For example, such as Figure 11 As shown, one or more computer-readable instructions 501 may be stored non-temporarily on the storage medium 500. For example, when the computer-readable instructions 501 are executed by a processor, one or more steps in the image processing method described above may be performed.

[0203] For example, the storage medium 500 can be used in the aforementioned electronic device 400. For example, the storage medium 500 may include the memory 402 in the electronic device 400.

[0204] For example, the description of storage medium 500 can be found in the description of memory 402 in the embodiment of electronic device 400, and the repeated parts will not be repeated.

[0205] Figure 12 This is a schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. The electronic device provided in this disclosure can be applied to an Internet system.

[0206] use Figure 12 The computer system provided herein can implement the functions of the image processing apparatus and / or electronic device involved in this disclosure. Such computer systems may include personal computers, laptops, tablets, mobile phones, personal digital assistants, smart glasses, smartwatches, smart rings, smart helmets, and any smart portable or wearable device. A specific system in this embodiment uses a functional block diagram to explain a hardware platform including a user interface. This computer device can be a general-purpose computer device or a purpose-specific computer device. Both types of computer devices can be used to implement the image processing apparatus and / or electronic device of this embodiment. The computer system may include any components necessary to implement the image processing described herein. For example, the computer system can be implemented by a computer device through its hardware, software programs, firmware, and combinations thereof. For convenience, Figure 12 Although only one computer device is shown in the figure, the computer functions related to the information required for image processing described in this embodiment can be implemented in a distributed manner by a set of similar platforms, thus distributing the processing load of the computer system.

[0207] like Figure 12As shown, the computer system may include a communication port 250 connected to a network for data communication. For example, the computer system can send and receive information and data through the communication port 250, enabling the computer system to exchange data via wireless or wired communication with other electronic devices. The computer system may also include a processor group 220 (i.e., the processor described above) for executing program instructions. The processor group 220 may consist of at least one processor (e.g., a CPU). The computer system may include an internal communication bus 210. The computer system may include different forms of program storage units and data storage units (i.e., the memory or storage media described above), such as a hard disk 270, read-only memory (ROM) 230, and random access memory (RAM) 240, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by the processor group 220. The computer system may also include an input / output component 260 for implementing input / output data flow between the computer system and other components (e.g., user interface 280, etc.).

[0208] Typically, the following devices can be connected to the input / output component 260: input devices such as touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices such as displays (e.g., LCD, OLED displays, etc.), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication interfaces.

[0209] Although Figure 12 A computer system with various devices is shown, but it should be understood that the computer system is not required to have all the devices shown, and alternatively, the computer system may have more or fewer devices.

[0210] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0211] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0212] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0213] The following points should be noted regarding this disclosure:

[0214] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0215] (2) For clarity, the thickness and dimensions of layers or structures are enlarged in the accompanying drawings used to describe embodiments of the invention. It will be understood that when an element such as a layer, film, region, or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element, or there may be intermediate elements present.

[0216] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0217] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. An image processing method, comprising: The camera device is used to capture the interactive actions performed by the interactive object on the recognized object, resulting in multiple frame images; The multiple frame images are blurred to obtain multiple processed images corresponding to the multiple frame images respectively, wherein the blurring process ensures that the interactive action object does not obstruct the displayed content of the recognition object in each processed image; Based on the multiple frame images, identify the interactive action of the interactive action object; Based on the multiple processed images and the interactive actions, execute the corresponding interactive operations. The identification of the interactive action of the interactive action object based on the plurality of frame images includes: For each frame image, determine the position of the key points of the interactive action object in the frame image; The interactive action is identified based on the position of the key points of the interactive action object in the multiple frame images.

2. The image processing method according to claim 1, wherein, The multiple frame images are blurred to obtain multiple processed images corresponding to the multiple frame images, including: Determine a reference image corresponding to the identified object, wherein the reference image includes at least a portion of the display content corresponding to the identified object; For each frame image: The occlusion region in the frame image, which is at least partly composed of the interactive action object, is determined using a segmentation model, wherein the at least part of the display content includes the display content in the occlusion region; The reference image and the occluded region are merged to obtain the first intermediate image corresponding to the frame image; The occluded area in the first intermediate image is blurred to obtain the processed image corresponding to the frame image.

3. The image processing method according to claim 2, wherein, Determining the reference image corresponding to the identified object includes: An image including all the display content of the identified object is captured as the reference image.

4. The image processing method according to claim 2, wherein, Determining the reference image corresponding to the identified object includes: The multiple frame images are sequentially processed to merge the backgrounds to obtain the reference image.

5. The image processing method according to claim 4, wherein, The reference image is obtained by sequentially performing background merging processing on the plurality of frame images, including: For the current frame image in the plurality of frame images that has undergone the background merging process, the segmentation model is used to determine the occlusion region in the current frame image that is at least partly composed of the interactive action object; Mark the pixels in the occluded region of the current frame image to obtain the second intermediate image corresponding to the frame image; Obtain a historical background merged image, wherein, in response to the current frame image being the first image to undergo the background merging process, the second intermediate image corresponding to the current frame image is used as the historical background merged image; The historical background merged image and the second intermediate image are matched. In response to the result of the matching process indicating that the identified object has not moved, The second intermediate image and the historical background merged image are then combined to obtain a merged image. If the proportion of pixels marked as occluded locations in the fused image is less than a certain percentage, the fused image is determined to be the reference image. Otherwise, the historical background merged image is updated to the fused image, and the background merging process continues for the next frame image; In response to the result of the matching process indicating that the identified object has moved, The historical background merged image is updated to the second intermediate image, and the background merging process continues for the next frame image.

6. The image processing method according to claim 5, wherein, The matching process between the merged historical background image and the second intermediate image includes: Determine the first histogram vector corresponding to the merged historical background image and the second histogram vector corresponding to the second intermediate image; The similarity between the first histogram vector and the second histogram vector is compared, and the result of the matching process is determined based on the similarity.

7. The image processing method according to claim 6, wherein, Comparing the similarity between the first histogram vector and the second histogram vector, and determining the result of the matching process based on the similarity, includes: Calculate the correlation coefficient between the first histogram vector and the second histogram vector; Calculate the Euclidean distance between the first histogram vector and the second histogram vector; The weighted average of the correlation coefficient and the Euclidean distance is calculated as the similarity. In response to the similarity being greater than a similarity threshold, the result of the matching process is determined to indicate that the identified object has not moved. In response to the similarity being less than the similarity threshold, the result of the matching process is determined to indicate that the identified object has moved.

8. The image processing method according to claim 5, wherein, The second intermediate image and the historical background merged image are then fused to obtain a merged image, including: Identify one or more pixels in the merged historical background image that are marked as occlusion locations; The pixel in the second intermediate image that corresponds to the one or more pixel points is determined as the replacement pixel point; Identify at least one pixel among the replacement pixels that is not marked as an occluded position; The values ​​of the pixels corresponding to the at least one pixel in the one or more pixels are replaced with the values ​​of the at least one pixel to obtain the fused image.

9. The image processing method according to claim 2, wherein, The reference image and the occluded region are merged to obtain a first intermediate image corresponding to the frame image, including: Add the display content corresponding to the occluded area in the frame image to the reference image to obtain the first intermediate image corresponding to the frame image.

10. The image processing method according to claim 2, wherein, Blurring the occluded region in the first intermediate image to obtain the processed image corresponding to the frame image includes: Increase the transparency of the occluded area in the first intermediate image to obtain the processed image corresponding to the frame image.

11. The image processing method according to claim 1, wherein, The interactive action object is a gesture action object. For each frame image, determine the position of the key points of the interactive action object in the frame image, including: Using a key point detection model, target key points in the gesture action object are identified; Based on the target key points, extract the detection region in the frame image that includes the target key points; Extract the feature position of at least one corner point within the detection area; Gaussian convolution is performed at each corner feature location to obtain the corner feature map corresponding to the frame image; The corner feature map and the corner feature map corresponding to the reference frame image are subjected to two-dimensional convolution to obtain the calculation result. The reference frame image is the frame image that is located before the frame image and adjacent to the frame image in the shooting order among the multiple frame images. Determine the position corresponding to the maximum value in the calculation result; The position of the target key point is corrected according to the position corresponding to the maximum value, so as to obtain the position of the key point of the gesture action object.

12. The image processing method according to claim 1, wherein, The interactive action objects include a first hand and a second hand. The key points of the interactive action object include a first key point corresponding to the first hand and a second key point corresponding to the second hand. Identifying the interactive action based on the position of the key points of the interactive action object in the multiple frame images includes: In response to the duration during which the positional changes of both the first keypoint and the second keypoint are less than the position threshold being greater than the time threshold: The interaction action is determined to be the first selected action, and The interaction area is determined based on the positions of the first key point and the second key point.

13. The image processing method according to claim 12, wherein, Based on the positions of the first key point and the second key point, the interaction area is determined, including: The outer region box is determined based on the positions of the first key point and the second key point, and the outer region box is used as the interaction area.

14. The image processing method according to claim 12, wherein, Based on the multiple processed images and the interactive actions, execute corresponding interactive operations, including: In response to the interaction action being the first selection action, Based on the multiple processed images, a text description of the content in the interactive area is obtained.

15. The image processing method according to claim 1, wherein, Identifying the interactive action based on the position of the key points of the interactive action object in the multiple frame images includes: In response to the key point of the interactive action object moving on the identified object and a stop action being detected: The interaction action is determined to be the second selected action, and Based on the position of the key points of the interactive action object, determine the movement trajectory of the key points of the interactive action object between the stop actions, and determine the interaction area based on the movement trajectory.

16. The image processing method according to claim 15, wherein, Based on the multiple processed images and the interactive actions, execute corresponding interactive operations, including: In response to the interaction action being the second selection action, Based on the multiple processed images, identify the content in the interactive area.

17. The image processing method according to any one of claims 2-10, wherein, The segmentation model includes a multi-scale module and a backbone network module. The multi-scale module includes a first sub-module, a second sub-module, and a third sub-module. The first submodule, the second submodule, and the third submodule are configured to extract features of different dimensions, wherein, The first submodule is configured to extract detailed features. The second submodule and the third submodule are configured to extract global features; The backbone network module is configured to fuse the features extracted by the first submodule, the second submodule, and the third submodule to obtain the output of the segmentation model.

18. The image processing method according to claim 17, wherein, The first submodule includes a 1*1 convolutional layer and a max pooling layer; The second submodule includes multiple first dilated convolutional layers and multiple deconvolutional layers, wherein the multiple first dilated convolutional layers are divided into multiple groups for grouped convolution; The third submodule includes multiple second dilated convolutional layers, wherein the multiple third dilated convolutional layers are divided into multiple groups for grouped convolution.

19. An image processing apparatus, comprising: The image acquisition module is configured to use a shooting device to capture the interactive actions performed by the interactive action object on the recognition object, and obtain multiple frame images; The processing module is configured to perform blurring processing on the plurality of frame images to obtain a plurality of processed images corresponding to the plurality of frame images respectively, wherein the blurring processing ensures that the interactive action object does not obstruct the display content in the recognition object in the processed image; The recognition module is configured to recognize the interactive action of the interactive action object based on the plurality of frame images; The execution module is configured to perform corresponding interactive operations based on the multiple processed images and the interactive actions. The identification module is configured as follows: For each frame image, determine the position of the key points of the interactive action object in the frame image; The interactive action is identified based on the position of the key points of the interactive action object in the multiple frame images.

20. An electronic device comprising an imaging device and an image processing unit, wherein, The shooting device is configured to capture the interactive actions performed by the interactive action object on the recognition object, thereby obtaining a video stream; The image processing unit is configured to receive the video stream, obtain the plurality of frame images from the video stream, and execute the image processing method according to any one of claims 1-18.

21. An electronic device, comprising: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the image processing method according to any one of claims 1-18.

22. A non-transitory computer-readable storage medium, wherein, The non-transitory computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the image processing method according to any one of claims 1-18.

Images