Image processing method and device and computer readable storage medium
By extracting and matching features to identify target objects in the image and generating detection frames for occlusion, the problem of insufficient flexibility and convenience of image processing in the prior art is solved, and flexible occlusion of designated targets in the image is achieved, improving user experience.
Patent Information
- Application Number
- CN202411104648.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-13
AI Technical Summary
The existing image processing technology is not flexible and convenient when occluding designated targets in the image, making it difficult to meet the flexible needs of users.
By obtaining search images and target images, feature extraction and matching are performed, target parts and objects are identified, and target detection boxes are generated based on the target objects specified by the user for occlusion, which supports users to select target objects to be blocked according to their needs.
It improves the convenience and flexibility of image processing, allows users to flexibly select and obstruct designated target objects in the image, and enhances the user experience of image processing.
Smart Images
Figure CN119131190A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and computer-readable storage medium. Background Art
[0002] With the development of science and technology and the improvement of people's living standards, image processing is involved in various fields. For example, in some scenes, it is necessary to block the specified target in the image. The inventor of this application found that the current image processing is not flexible enough and the convenience needs to be improved. Summary of the invention
[0003] The present application provides an image processing method, device and computer-readable storage medium, which can improve the flexibility and convenience of image processing.
[0004] A first aspect of an embodiment of the present application provides an image processing method, which includes: acquiring a search image and at least one target image, the target image including a target part, the target parts in different target images belonging to different target objects; performing feature extraction on the search image to obtain search image features; performing feature extraction on each of the target images respectively to obtain a first image feature of each target image; identifying the target part in the search image according to the first image feature of each target image and the search image feature; identifying each of the target objects in the search image, and determining a target detection frame of the target object for each of the target objects in the search image; and for each of the target objects in the search image, in response to the target detection frame of the target object matching with any of the target parts in the search image, masking the target object in the search image.
[0005] A second aspect of an embodiment of the present application provides an image processing device, which includes a processor, a memory, and a communication circuit. The processor is coupled to the memory and the communication circuit, respectively. Program data is stored in the memory. The processor implements the steps in the above method by executing the program data in the memory.
[0006] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the steps in the above method.
[0007] The beneficial effect is that the method of the present application can block the target object specified by the user in the search image, and the user can choose to specify the target object to be blocked at any time according to needs, so the method of the present application can improve the convenience and flexibility of image processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:
[0009] Figure 1 It is a flowchart of an implementation method of the image processing method of the present application;
[0010] Figure 2 yes Figure 1 Schematic diagram of the process of step S140;
[0011] Figure 3 yes Figure 2 Schematic diagram of the process of step S141;
[0012] Figure 4 It is a schematic diagram of the process framework of the present application for performing sparse processing on search image features;
[0013] Figure 5 It is a simple structural schematic diagram of an image processing model in one embodiment of the present application;
[0014] Figure 6 It is a structural schematic diagram of an embodiment of the image processing device of the present application;
[0015] Figure 7 is a structural schematic diagram of another embodiment of the image processing device of the present application;
[0016] Figure 8 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION
[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0018] It should be noted that the terms "first" and "second" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.
[0019] First of all, it should be noted that the image processing method of the present application is executed by an image processing device, and the image processing device can be any device with image processing capabilities such as a mobile phone, a computer, etc.
[0020] See also Figure 1 , Figure 1 : is a flowchart of an embodiment of an image processing method of the present application, the image processing method comprising:
[0021] S110: Acquire a search image and at least one target image, where the target image includes a target part, and the target parts in different target images belong to different target objects.
[0022] In one embodiment, a search image is obtained from a video stream, wherein each image in the video stream may be used as a search image, or some images in the video stream may be used as search images. When there are multiple search images, steps S110-S160 are performed for each search image.
[0023] The target part is a part on the target object specified by the user. The target object specified by the user in the search image can be blocked by the image processing method of the present application. For example, the target object specified by the user in the search image can be mosaicked. The number of target objects specified by the user can be one or more, which is set by the user according to the needs. At the same time, the target image is an image of the target part of the target object, that is, the target image only includes the target part of the target object, rather than the entire target object.
[0024] In an application scenario, when a user wants to specify to block several target objects in a search image, the user inputs target images corresponding to the several target objects on an image processing device.
[0025] S120: Extract features from the search image to obtain search image features.
[0026] Specifically, any feature extraction method may be used to extract features from the search image to obtain search image features.
[0027] S130: Extract features from each target image respectively to obtain a first image feature of each target image.
[0028] Specifically, any feature extraction method may be used to extract features from each target image to obtain the first image feature of each target image.
[0029] S140: Identify a target part in the search image according to the first image feature of each target image and the search image feature.
[0030] Specifically, each target part in the search image can be identified according to the first image feature of each target image and the search image feature.
[0031] It should be noted that the target part in the search image identified at this time may only include the part of the target object specified by the user. This is because the search image only includes the part of the target object specified by the user.
[0032] S150: Identify each target object in the search image, and determine a target detection frame of each target object in the search image.
[0033] Specifically, all target objects in the search image are first identified, and for each identified target object, a target detection frame of the target object is generated. In one embodiment, the minimum bounding rectangle of the target object can be used as the target detection frame of the target object.
[0034] S160: For each target object in the search image, in response to a target detection frame of the target object matching with any target part in the search image, masking the target object in the search image.
[0035] Specifically, for each target object identified in the search image, the following steps are performed respectively:
[0036] The target detection frame of the target object is matched with the target part in the search image in sequence. Once the target detection frame of the target object is successfully matched with any target part in the search image, it means that the target object is the target object specified by the user. Finally, the target object is blocked in the search image, and the target object cannot be displayed when the search image is displayed subsequently. If the target detection frame of the target object cannot be matched with each target part in the search image, it means that the target object is not the target object specified by the user, and there is no need to block the target object. The target object will be displayed normally when the search image is displayed subsequently.
[0037] The process of determining whether a target detection frame of the target object matches a target part in the search image may include determining whether a position and a size of the target detection frame of the target object matches a position and a size of the target part in the search image.
[0038] For ease of understanding, the above scheme is explained here with examples:
[0039] Assuming that the user wants to block target objects A and B in the video stream, the user can pre-input target image 1 and target image 2 on the image processing device, where target image 1 includes the target part of target object A and target image 2 includes the target part of target object B.
[0040] After receiving target image 1 and target image 2, the image processing device extracts features of the search image, target image 1 and target image 2 in the video stream respectively to obtain the search image features, the first image features corresponding to target image 1 and the first image features corresponding to target image 2.
[0041] Then, the image processing device identifies the target part included in the search image based on the search image features, the first image features corresponding to target image 1 and the first image features corresponding to target image 2. It is assumed that at this time, it is identified that the search image only includes one target part, and the target part is defined as target part C.
[0042] And, the image processing device recognizes the search image and recognizes all target objects in the search image. At this time, all the recognized target objects may include target object A and target object B, or may only include one of target object A and target object B, or may not include target object A and target object B. Here, it is assumed that four target objects are recognized in the search image, and the four target objects are respectively recorded as target object a, target object b, target object c, and target object d. Then, a target detection frame a1 of target object a, a target detection frame b1 of target object b, a target detection frame c1 of target object c, and a target detection frame d1 of target object d are generated.
[0043] Next, for target object a, determine whether the target detection frame a1 of target object a matches the target part C. If the match is successful, it means that target object a is one of target object A and target object B, then target object a is blocked in the search image, otherwise target object a is not processed; the above steps are performed for target object b, target object c and target object d.
[0044] Through the method of the present application, the user-specified target object in the search image can be blocked, and the user can choose to specify the target object to be blocked at any time according to needs. Therefore, the method of the present application can improve the convenience and flexibility of image processing.
[0045] In one embodiment, when the target object is blocked in the search image, only the inside of the target object outline may be blocked, rather than the entire target detection frame of the target object. That is to say, the area outside the target object in the target detection frame will not be blocked. In another embodiment, when the target object is blocked in the search image, the entire target detection frame of the target object may be blocked. In short, the present application does not limit the blocked area, as long as the target object can be blocked.
[0046] In one embodiment, for each target object in the search image, a target detection frame of the target object is Hungarian matched with each target part in the search image to determine whether the target detection frame of the target object can match any target part in the search image, that is, the target detection frame of the target object in the search image is matched with the target part in the search image using the Hungarian matching algorithm to determine whether the two can be successfully matched. In other embodiments, other matching algorithms can also be used for matching, and this application does not limit the matching process.
[0047] In one embodiment, the search image may be input into a first visual feature encoder, the visual features of the search image may be extracted using the first visual feature encoder, and the visual features may be flattened in the spatial dimension to obtain aggregated search image features. The search image features are denoted as F v , F v ∈R 1×(H×W)×C , where H, W, and C represent the width, height, and number of channels of the search image features, respectively, and F v ∈R 1×(H×W)×C It is also characterized that the search image feature includes (H×W) first feature blocks (ie, the number of the first feature blocks is the product of H and W), and the dimension of the first feature block is C (the first feature block includes C values).
[0048] In this embodiment, the same as the feature extraction process of the search image, multiple target images are input into the second visual feature encoder, so that the second visual feature encoder outputs the first image feature of each target image. Assuming that the number of target objects specified by the user is m, the number of target images is also m, and the first image features of the m target images are respectively recorded as F f1 、F f2 ,…,F fi ,…,F fm , 1≤i≤m, F fi ∈R 1×h×w×C , h, w, and C represent the width, height, and number of channels of the first image feature, respectively. The number of channels of the search image feature is equal to the number of channels of the first image feature. The first visual feature encoder and the second visual feature encoder can be the same encoder or different encoders, without limitation. The first visual feature encoder and the second visual feature encoder can be any backbone network with strong feature extraction capability, such as ViT, ResNet, MobileNet and other series networks, without limitation.
[0049] In one embodiment, see Figure 2 The process of step S140 identifying the target part in the search image includes:
[0050] S141: Perform sparse processing on the search image features to obtain sparse image features.
[0051] Among them, the number of feature blocks in the sparse image features is less than the number of feature blocks in the search image features.
[0052] S142: Determine whether the number of feature blocks in the sparse image features is greater than a number threshold.
[0053] If the judgment result is yes, execute step S143, otherwise execute step S144.
[0054] S143: Using the sparse image features as search image features.
[0055] After executing step S143, return to execute step S141.
[0056] S144: Identify a target part in the search image according to the sparse image features and the first image features of each target image.
[0057] Specifically, before identifying the target part, the search image features are first subjected to sparse processing to reduce the number of feature blocks in the search image features, thereby obtaining sparse image features. The sparse processing process can reduce the amount of algorithm processing for subsequent recognition, thereby improving the processing efficiency of the entire algorithm. The process of sparse processing the search image features can be to discard some feature blocks in the search image features, or other methods, as described below.
[0058] At the same time, before executing step S144, it is determined whether the number of feature blocks in the sparse image features is greater than the quantity threshold. If it is, the sparse processing is continued to ensure that the number of feature blocks in the final sparse image features does not exceed the quantity threshold, thereby further reducing the subsequent algorithm processing amount.
[0059] It should be noted that, in other implementations, step S142 may not be executed before step S144 is executed. In this case, step S144 is directly executed after step S141.
[0060] See also Figure 3 In one embodiment, step S141 includes:
[0061] S1411: extract common features based on the first image features of each target image to obtain common feature blocks, wherein the dimensions of the common feature blocks are equal to the dimensions of the first feature blocks.
[0062] Specifically, the common feature block represents the common features in multiple target images. The common feature block is defined as token′, token′∈R 1×C , indicating that the dimension of the common feature block token′ is equal to that of the first feature block.
[0063] S1412: Determine the similarity between each first feature block and the common feature block respectively.
[0064] Specifically, the higher the similarity between the first feature block and the common feature block, the more similar the first feature block and the common feature block are.
[0065] S1413: Fusing the other first feature blocks except for a preset number of first feature blocks with the highest corresponding similarities to obtain fused feature blocks.
[0066] Specifically, after obtaining the similarity between each first feature block and the common feature, the first feature blocks can be sorted in order from large to small according to the corresponding similarity, and then a preset number of first feature blocks with the highest corresponding similarity are selected, and the other unselected first feature blocks are fused to obtain a fused feature block.
[0067] The preset number n can be preset according to the requirements. For example, if n is set to 10, 10 first feature blocks with the highest corresponding similarities are selected, and the remaining first feature blocks are fused to obtain a fused feature block.
[0068] The preset number n can also be obtained according to a preset ratio. For example, a ratio p is preset. L is the total number of the first feature blocks, symbol Indicates rounding down.
[0069] In one embodiment, the above-mentioned fusion processing includes: performing weighted summation processing on other first feature blocks except a preset number of first feature blocks with the highest corresponding similarity to obtain a fused feature block, wherein the weight of the first feature block is equal to the similarity corresponding to the first feature block.
[0070] Specifically, the fusion feature block is recorded as F v,fuse , the other first feature blocks except the preset number of first feature blocks with the highest similarity are recorded as F v,i , 1≤i≤Ln, the first feature block is denoted by F v,i The corresponding similarity is score i , calculate the fusion feature block F according to the following formula v,fuse :
[0071]
[0072] In other implementations, the above fusion process may also be: summing up the first feature blocks except for a preset number of first feature blocks with the highest corresponding similarities to obtain a fused feature block.
[0073] S1414: A preset number of first feature blocks with the highest corresponding similarity are serially processed with the fused feature block to obtain sparse image features.
[0074] Specifically, the selected n first feature blocks are connected in series with the fused feature block obtained in step S1413 to obtain a sparse image feature. At this time, the sparse image feature includes (n+1) feature blocks.
[0075] In one embodiment, after the first feature blocks with the highest corresponding similarity are connected in series with the fused feature blocks, the features obtained in series are passed through a feedforward neural network FFN and Add&Norm to obtain sparse image features. Wherein, the number of feature blocks in the feature is not changed by passing through a feedforward neural network FFN and Add&Norm, that is, the sparse image feature still includes (n+1) feature blocks. Wherein, Add&Norm means performing a residual on the feature and then performing a layer normalization operation.
[0076] It can be understood that, compared with discarding the first feature blocks other than the preset number of first feature blocks with the highest similarity, the above scheme processes the preset number of first feature blocks with the highest similarity in series with the fused feature blocks to obtain sparse image features, which can reduce the impact of background information loss on recognition and improve recognition accuracy. In other words, the above-mentioned sparse process can reduce the number of feature blocks and improve the efficiency of the algorithm without discarding background information, so it can take into account both computational efficiency and algorithm accuracy, and ensure the accuracy of the final result.
[0077] In one embodiment, the sparse image features finally obtained are recorded as F v,sparse , F v,sparse ∈R w×C , w represents the number of feature blocks in the sparse image features, and the dimension of the feature blocks in the sparse image features is also C.
[0078] In one embodiment, the process of obtaining the common feature blocks in step S1411 includes:
[0079] S14111: Fusing the first image features of each target image to obtain a second image feature, wherein the second image feature includes a second number of second feature blocks, the dimension of the first feature block is equal to the dimension of the second feature block, and the second number is equal to the number of target images.
[0080] Specifically, assuming that the number of target objects specified by the user is m, and the number of target images is also m, the second image feature can be recorded as F f , F f ∈R 1×m×C , indicating that the second image feature includes m second feature blocks, and the dimension of the second feature block is also C, which is equal to the dimension of the first feature block.
[0081] In one embodiment, the specific process of step S14111 includes: firstly using the maximum pooling layer to respectively pool the first image feature F f1 , the first image feature F f2 , ..., the first image feature F fi , ..., the first image feature F fm Perform feature aggregation to obtain m aggregated image features, and finally connect the m aggregated image features in series to obtain the second image feature F f .
[0082] S14112: Extract common features from the second image features to obtain common feature blocks.
[0083] Specifically, after obtaining the second image feature F fAfterwards, common features are extracted to obtain common feature blocks token′.
[0084] In one embodiment, step S14112 specifically includes: obtaining a first embedded feature block token, and determining a first embedded feature block token based on the first embedded feature block token and the second image feature F f , get the common feature token′. token∈R 1×C It indicates that the dimension of the first embedded feature block is C, which is equal to the dimensions of the first feature block and the second feature block. The first embedded feature token can be randomly initialized and optimized and learned in the subsequent network training process.
[0085] Among them, the common feature block token′ can be obtained according to the following formula:
[0086]
[0087] in, is the second image feature F f The transposed matrix of , softmax() indicates a softmax operation, Linear() indicates a fully connected layer operation, and LN() indicates a layer normalization operation.
[0088] In one embodiment, the process of respectively determining the similarity between each first feature block and the common feature block in S1412 includes:
[0089] Determine the similarity matrix score according to the following formula:
[0090]
[0091] Among them, score∈R 1×L , score includes the similarity between each of the L first feature blocks and the common feature block, To search for image features F v The transposed matrix of , softmax() represents the softmax operation, and C is the dimension of the first feature block.
[0092] After obtaining the score, the similarity between each first feature block and the common feature block token′ can be extracted from the score.
[0093] For a better understanding, the following combination Figure 4 , details the specific process of obtaining sparse image features in a specific example:
[0094] (11) Search image feature F v Perform feature enhancement processing, wherein the specific process of feature enhancement processing includes: searching for image feature F vThe multi-head attention mechanism and Add&Norm processing are performed in sequence. v The step of performing enhancement processing is not necessary and may not be performed. For the sake of convenience, the enhanced search image features are still referred to as search image feature blocks below.
[0095] (12) Based on the first embedded feature block token and the second image feature F f Common features are extracted to obtain common feature blocks token′.
[0096] (13) Based on the cross-attention mechanism, the similarity matrix is calculated to obtain the similarity between each first feature block and the common feature block.
[0097] (14) According to the similarity matrix, multiple first feature blocks in the search image features are selected: a preset number of first feature blocks with the highest similarity are retained, and other first feature blocks are fused to obtain fused feature blocks. After the selection process, the preset number of first feature blocks with the highest similarity and the fused feature blocks are connected in series.
[0098] (15) The concatenated features are passed through a feed-forward neural network FFN and Add&Norm to obtain sparse image features.
[0099] In one embodiment, step S150 uses the following method to identify each target object in the search image:
[0100] (21) Set the segmentation embedding feature F p,ins , F p,ins ∈R 100×C , where 100 means that the network can detect up to 100 target objects in an image, and the segmentation embedding feature F p,ins Random initialization is used to optimize learning during network training.
[0101] (22) Embed the segmentation into features F p,ins After a layer of standard Transformer-Encoder network for feature interaction learning, we get F′ p,ins , F′ p,ins ∈R 100×C ;
[0102] (23) Use the reshape operation to reshape the search image feature F v reshape into F′ v , F′ v ∈R 1×H×W×C , and then use the PixelDecoder network to convert F′ vGradually perform upsampling enhancement decoding to obtain high-resolution image features F v,high , F v,high ∈R 1×(b×H)×(b×W)×C , where b represents the upsampling multiple of the feature map after the PixelDecoder network.
[0103] (24) F′ p,ins (F′ p,ins ∈R 100×C ) and search image features F v (F v ∈R 1×(H×W)×C ) is input into the standard TransformerDecoder network to calculate the conditional convolution kernel parameters F p,cond (F p,cond ∈R 100×C );
[0104] (25) Finally, the high-resolution image feature F v,high The matrix is multiplied by the conditional convolution kernel parameters predicted by the network to obtain the segmentation result mask (mask∈R (b×H)×(b×W)×100 ), each feature channel in the mask represents the instance segmentation result of a target object, and up to 100 target objects in the image can be segmented.
[0105] After the above steps, each target object in the search image can be identified.
[0106] It should be noted that the present application does not limit the specific process of identifying the target object in the search image, and any method can be used for identification.
[0107] In one embodiment, step S142 includes: using the second image feature as a query, and the sparse image feature F v,sparse The key and value are input into the target part detection network composed of multiple standard Transformer-Decoders to detect the target part, and the detection box and classification results of each target part in the search image are obtained.
[0108] In order to better understand the method of the present application, Figure 5 The specific process of an image processing method in an example is described in detail below. In the method, an image processing model is used to participate in image processing. The image processing model includes a visual feature encoder, a sparse network, an object segmentation network, and a target part detection network. The image processing method includes:
[0109] (31) using a visual feature encoder to participate in the processing, obtaining a search image feature of the search image and a first image feature of each target image;
[0110] (32) fusing the first image features of all target images to obtain second image features;
[0111] (33) inputting the search image feature, the first embedded feature, and the second image feature into a sparse network to perform sparse processing on the search image feature to obtain a sparse image feature;
[0112] (34) inputting the sparse image features and the second image features into a target part detection network to identify the target part, thereby obtaining the target part in the search image;
[0113] (35) Inputting the segmentation embedding features and the search image features into the object segmentation network to obtain the target object in the search image;
[0114] (36) Matching the target part in the search image with the target detection frame of the target object in the search image, and masking the successfully matched target object in the search image.
[0115] See also Figure 6 , Figure 6 1 is a schematic diagram of the structure of an embodiment of the image processing device of the present application. The image processing device 200 includes a processor 210, a memory 220 and a communication circuit 230. The processor 210 is coupled to the memory 220 and the communication circuit 230 respectively. The memory 220 stores program data. The processor 210 implements the steps in any of the above-mentioned implementation methods by executing the program data in the memory 220. The detailed steps can be found in the above-mentioned implementations and will not be repeated here.
[0116] The image processing device 200 may be any device with image processing capability, such as a computer or a mobile phone, and is not limited here.
[0117] See also Figure 7 , Figure 7 The image processing device 300 includes an acquisition module 310 , a first feature extraction module 320 , a second feature extraction module 330 , a first recognition module 340 , a second recognition module 350 , and an occlusion module 360 .
[0118] The acquisition module 310 is used to acquire a search image and at least one target image, wherein the target image includes a target part, and the target parts in different target images belong to different target objects.
[0119] The first feature extraction module 320 is connected to the acquisition module 310 and is used to extract features from the search image to obtain search image features.
[0120] The second feature extraction module 330 is connected to the acquisition module 310 and is used to extract features from each target image to obtain the first image feature of each target image.
[0121] The first recognition module 340 is connected to the first feature extraction module 320 and the second feature extraction module 330, and is used to recognize the target part in the search image according to the first image feature of each target image and the search image feature.
[0122] The second recognition module 350 is connected to the acquisition module 310 and is used to recognize each target object in the search image and determine a target detection frame of the target object for each target object in the search image.
[0123] The occlusion module 360 is connected to the first recognition module 340 and the second recognition module 350, and is used to occlude each target object in the search image in response to the target detection frame of the target object matching with any target part in the search image.
[0124] The image processing device 300 may be any device with image processing capability, such as a computer or a mobile phone, and is not limited here. The image processing device 300 performs the method steps in any of the above embodiments when working. The detailed method can be found in the above related content and will not be repeated here.
[0125] See also Figure 8 , Figure 8 The computer-readable storage medium 400 stores a computer program 410, which can be executed by a processor to implement the steps in any of the above methods.
[0126] Among them, the computer-readable storage medium 400 can specifically be a device that can store the computer program 410, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or it can also be a server that stores the computer program 410. The server can send the stored computer program 410 to other devices for execution, or it can also run the stored computer program 410 by itself.
[0127] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. An image processing method, It is characterized in that The method comprises: Acquire a search image and at least one target image, wherein the target image includes a target part, and the target parts in different target images belong to different target objects; Extracting features from the search image to obtain search image features; Performing feature extraction on each of the target images respectively to obtain a first image feature of each of the target images; identifying the target part in the search image according to the first image feature and the search image feature of each of the target images; Identify each of the target objects in the search image, and determine a target detection frame of the target object for each of the target objects in the search image; For each of the target objects in the search image, in response to the target detection frame of the target object matching with any of the target parts in the search image, the target object is blocked in the search image.
2. The method according to claim 1, It is characterized in that The step of identifying the target part in the search image according to the first image feature and the search image feature of each of the target images comprises: Performing a sparse processing on the search image feature to obtain a sparse image feature, wherein the number of feature blocks in the sparse image feature is less than the number of feature blocks in the search image feature; The target part in the search image is identified according to the sparse image features and the first image features of each of the target images.
3. The method according to claim 2, It is characterized in that The search image feature includes a plurality of first feature blocks; The step of performing sparse processing on the search image features to obtain sparse image features includes: Extracting common features based on the first image features of each of the target images to obtain common feature blocks, wherein the dimensions of the common feature blocks are equal to the dimensions of the first feature blocks; respectively determining the similarity between each of the first feature blocks and the common feature block; Fusing the other first feature blocks except the preset number of first feature blocks corresponding to the highest similarity to obtain fused feature blocks; The first feature blocks corresponding to the preset number of the highest similarities are connected in series with the fused feature blocks to obtain the sparse image features.
4. The method according to claim 3, It is characterized in that The step of extracting common features based on the first image features of each of the target images to obtain common feature blocks includes: fusing the first image features of each of the target images to obtain second image features, wherein the second image features include a second number of second feature blocks, the dimensions of the first feature blocks are equal to the dimensions of the second feature blocks, and the second number is equal to the number of the target images; Common features are extracted from the second image features to obtain the common feature blocks.
5. The method according to claim 4, It is characterized in that The step of extracting common features from the second image features to obtain the common feature blocks comprises: Acquire a first embedded feature block, where the first embedded feature block has the same dimension as the second feature block; The common feature block is obtained according to the first embedded feature block and the second image feature.
6. The method according to claim 5, It is characterized in that The step of obtaining the common feature block according to the first embedded feature block and the second image feature comprises: The common feature block is obtained according to the following formula: Among them, token ′ is the common feature block, F f is the second image feature, is the transposed matrix of the second image feature, C is the dimension of the first embedded feature block, softmax() represents a softmax operation, Linear() represents a fully connected layer operation, and LN() represents a layer normalization operation.
7. The method according to claim 3, It is characterized in that The step of respectively determining the similarity between each of the first feature blocks and the common feature block comprises: The similarity matrix is calculated according to the following formula: Among them, score is the similarity matrix, softmax() represents the softmax operation, token ′ is the common feature block, is the transposed matrix of the search image feature, and C is the dimension of the first feature block; The similarity between each of the first feature blocks and the common feature blocks is extracted from the similarity matrix.
8. The method according to claim 3, It is characterized in that The step of fusing the other first feature blocks except the preset number of first feature blocks corresponding to the highest similarity to obtain a fused feature block comprises: The other first feature blocks except a preset number of the first feature blocks corresponding to the highest similarity are weighted summed to obtain the fused feature block, wherein the weight of the first feature block is equal to the similarity corresponding to the first feature block.
9. The method according to claim 2, It is characterized in that Before identifying the target part in the search image according to the sparse image features and the first image features of each of the target images, the method further includes: Determining whether the number of feature blocks in the sparse image features is greater than a number threshold; In response to the number of feature blocks in the sparse image feature being greater than the number threshold, the sparse image feature is used as the search image feature, and the step of performing sparse processing on the search image feature to obtain the sparse image feature is returned to be executed; In response to the number of feature blocks in the sparse image features being not greater than the number threshold, the step of identifying the target part in the search image based on the sparse image features and the first image features of each of the target images is performed.
10. The method according to claim 1, It is characterized in that The method further comprises: For each target object in the search image, Hungarian matching is performed on the target detection frame of the target object and each target part in the search image to determine whether the target detection frame of the target object can match any target part in the search image.
11. An image processing device, It is characterized in that The image processing device includes a processor, a memory and a communication circuit, the processor is coupled to the memory and the communication circuit respectively, the memory stores program data, and the processor implements the steps in the method as described in any one of claims 1-10 by executing the program data in the memory.
12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the steps in the method according to any one of claims 1 to 10.