Image occlusion detection method and device, and storage medium

Abstract

The invention provides an image occlusion detection method, system and device and a storage medium, and the method comprises the steps: obtaining a target detection image which comprises a reference object located in a reference region and a target object located in a target region; when the target object is completely imaged, the value of a first pixel channel corresponding to the target area in the detection image is smaller than the value of a first pixel channel corresponding to the reference area in the detection image; determining a first value of a first pixel channel corresponding to the reference area in the target detection image; determining a second value of a first pixel channel corresponding to the target object in the target detection image; and determining the shielding condition of the target object according to the numerical relationship between the first value and the second value. The method does not depend on a specific type of image or target object, a complex image processing algorithm can be avoided only by setting a reasonable reference area and paying attention to the difference of pixel channel values, calculation is simple, and universality is high.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image occlusion detection method, device and storage medium. Background Art

[0002] With the rapid development of computer vision technology, refrigeration equipment such as freezers or refrigerators can automatically identify the goods inside by capturing two-dimensional static images through cameras. This technology not only realizes the automated management of inventory, but also improves the timeliness or accuracy of replenishment.

[0003] However, in actual applications, the complexity of the refrigerator's internal environment and the dynamic nature of the usage scenarios bring many challenges to product recognition. In particular, when the refrigerator door is opened by the user, due to accidental occlusion by the human body or other objects, the image captured by the camera often contains non-product information, such as the outline of hands, faces or other body parts. These occlusions will significantly affect the accuracy and efficiency of image recognition.

[0004] In addition, most existing refrigerators upload the original images collected by the camera directly to a remote server for processing. Although this processing method simplifies the design complexity of refrigerators and other refrigeration equipment, it has obvious shortcomings in terms of network latency and bandwidth limitations between the refrigeration equipment and the cloud. Therefore, it is an urgent problem to filter out detection images with abnormal occlusions to avoid misjudgment and interference in product identification caused by occlusions. Summary of the invention

[0005] One of the purposes of the present invention is to provide an image occlusion detection method to solve the technical problems in the prior art that image occlusion detection relies on manual inspection or external measuring equipment, and has low detection efficiency and large errors.

[0006] In order to achieve one of the above-mentioned purposes of the invention, the present invention provides an image occlusion detection method, comprising: acquiring a target detection image, the target detection image including a reference object located in a reference area, and a target object located in a target area; when the target object is fully imaged, the value of the first pixel channel corresponding to the target area in the detection image is less than the value of the first pixel channel corresponding to the reference area in the detection image; determining a first value of the first pixel channel corresponding to the reference area in the target detection image; determining a second value of the first pixel channel corresponding to the target object in the target detection image; and determining the occlusion condition of the target object based on the numerical relationship between the first value and the second value.

[0007] As a further improvement of an embodiment of the present invention, the target area includes a first target area, the first value forms a first matrix, including a number of reference brightness matrix elements corresponding to a number of pixel points in the reference area; the second value forms a second matrix, including a number of target brightness matrix elements corresponding to a number of pixel points in the first target area; determining the occlusion condition of the target object based on the numerical relationship between the first value and the second value includes: calculating the difference between the corresponding matrix elements of the second matrix and the first matrix, and summing all the differences to obtain a first measurement result; determining the occlusion condition of the first target area based on the first measurement result.

[0008] As a further improvement of an embodiment of the present invention, determining the occlusion condition of the first target area based on the first measurement result includes: judging whether the first measurement result is greater than 0; if so, determining that there is occlusion in the first target area of ​​the target detection image, and setting all elements of the second matrix to 1; if not, determining that there is no occlusion in the first target area of ​​the target detection image, and setting all elements of the second matrix to 0.

[0009] As a further improvement of one embodiment of the present invention, determining the first value of the first pixel channel corresponding to the reference area in the target detection image includes: determining the first value based on the maximum value of the first pixel channel pixel values ​​corresponding to several reference objects in the reference area in the target detection image.

[0010] As a further improvement of one embodiment of the present invention, determining the second value of the first pixel channel corresponding to the target object in the target detection image includes: determining a convolution window based on the first value, and the convolution window is used to scan the target detection image; based on the convolution window, determining and extracting the second value of the first pixel channel corresponding to the first target area in the target detection image.

[0011] As a further improvement of an embodiment of the present invention, before obtaining the target detection image, the method includes: obtaining an initial detection image, and segmenting the initial detection image to determine a number of sub-image blocks; determining corresponding first eigenvectors and second eigenvectors based on the several sub-image blocks, the first eigenvector being used to determine content information of the sub-image block, and the second eigenvector being used to determine spatial position information of the sub-image block in the initial detection image; and fusing the first eigenvector and the second eigenvector to determine the shallow image feature vector.

[0012] As a further improvement of an embodiment of the present invention, the fusion of the shallow image feature vector and the deep image feature vector to determine the first fused feature vector includes: according to the shallow image feature vector, using a multi-stage downsampling feature extraction operation to determine a corresponding number of third feature vectors; using a multi-stage upsampling feature extraction operation on the number of third feature vectors to determine a corresponding number of fourth feature vectors; fusing the third feature vectors and the corresponding fourth feature vectors to determine a number of second fused feature vectors; determining the target detection image according to the first fused feature vector includes: according to the number of second fused feature vectors, using a decoding operation to determine the target detection image.

[0013] As a further improvement of an embodiment of the present invention, the extraction and determination of the shallow image feature vector and the deep image feature vector of the initial detection image includes: segmenting the initial detection image to determine a number of sub-image blocks; determining corresponding first feature vectors and second feature vectors based on the several sub-image blocks, the first feature vector being used to determine content information of the sub-image block, and the second feature vector being used to determine spatial position information of the sub-image block in the initial detection image; and fusing the first feature vector and the second feature vector to determine the shallow image feature vector corresponding to the initial detection image.

[0014] As a further improvement of an embodiment of the present invention, the extraction and determination of the shallow image feature vector and the deep image feature vector of the initial detection image includes: based on a multi-stage feature extraction structure, determining the corresponding deep image feature vector according to the shallow image feature vector.

[0015] As a further improvement of an embodiment of the present invention, the multi-stage feature extraction structure is based on which the corresponding deep image feature vector is determined according to the shallow image feature vector of the initial detection image, including: according to the shallow image feature vector, a multi-stage downsampling feature extraction operation is used to determine the corresponding third feature vectors; and a multi-stage upsampling feature extraction operation is used on the third feature vectors to determine the corresponding deep image feature vector.

[0016] To achieve one of the above-mentioned purposes of the invention, the present invention also provides a refrigeration device, comprising: a memory for storing executable instructions; a processor for implementing the steps of any of the above-mentioned image occlusion detection methods when running the executable instructions stored in the memory.

[0017] In order to achieve one of the above-mentioned purposes of the invention, the present invention also provides a computer-readable storage medium, including: at least one processor; a memory, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps of the image occlusion detection method are executed.

[0018] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0019] The present invention adopts an image occlusion detection method. By setting a reference area and a target area, and comparing and analyzing the difference in pixel values ​​of the reference area and the target area in the first pixel channel in the target detection image, it can automatically and quickly detect whether the target object is occluded, and has high judgment accuracy. This method reduces the dependence on manual judgment, does not require the use of other detection equipment, avoids measurement errors, and improves detection efficiency and accuracy; in addition, the method does not rely on a specific type of image or target object. As long as a reasonable reference area is set and attention is paid to the difference in pixel channel values, complex image processing algorithms can be avoided, and the calculation is simple and the versatility is strong. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the steps of an image occlusion detection method in one embodiment of the present invention.

[0021] Figure 2 It is a schematic diagram of a freezer row photographed when taking items from a three-dimensional freezer in one embodiment of the present invention.

[0022] Figure 3 It is a schematic diagram of steps before step S1 in one embodiment of the present invention.

[0023] FIG. 4( a ) is a schematic diagram of step S01 in a specific implementation of an embodiment of the present invention.

[0024] FIG. 4( b ) is a schematic diagram of step S01 ′ in a specific implementation of an embodiment of the present invention.

[0025] Figure 5 It is a schematic diagram of step S02 in a specific embodiment of an embodiment of the present invention.

[0026] Figure 6 It is a structural diagram of a visual converter model in a specific embodiment of an embodiment of the present invention.

[0027] FIG. 7( a ) is a schematic diagram of depth information from a baffle of a three-dimensional refrigerator to a camera in a specific embodiment of an embodiment of the present invention.

[0028] FIG. 7( b ) is a grayscale schematic diagram of a target detection image including a reference object according to an embodiment of the present invention.

[0029] Figure 8 2 is a schematic diagram of step S3 in one embodiment of the present invention.

[0030] Fig. 9 2 is a schematic diagram of step S4 in one embodiment of the present invention.

[0031] FIG. 10( a ) is a schematic diagram of a calculation process for calculating whether an image for target detection is occluded in one embodiment of the present invention.

[0032] FIG. 10( b ) is a schematic diagram of calculating grayscale of an image with occlusion for target detection according to an embodiment of the present invention.

[0033] FIG. 10( c ) is a schematic diagram of calculating the grayscale of a target detection image without occlusion according to an embodiment of the present invention.

[0034] Fig.11 It is a flowchart of an image occlusion detection method in one embodiment of the present invention.

[0035] Fig.12 A schematic structural diagram of a refrigeration device in one embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0037] It should be noted that the term "comprises" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, the terms "first", "second", "third", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance.

[0038] like Figure 1 As shown, an embodiment of the present invention provides an image occlusion detection method.

[0039] Image occlusion refers to the situation in which the target object or target area is partially or completely hidden in the image due to the presence of certain objects, obstacles or other factors during the image processing or visual perception process, making it impossible for the observer to directly see these occluded parts. For example, when a user or a replenisher opens the refrigerator door, the photos of the refrigerator display taken by the user may be blocked by human body, which affects the accuracy of product recognition.

[0040] Image occlusion detection is used to identify and determine whether there is occlusion in the target detection image, and can further analyze the degree, location and characteristics of the occluded object.

[0041] like Figure 1 As shown, an image occlusion detection method provided in one embodiment of the present invention includes the following steps.

[0042] Step S1, acquiring a target detection image, the target detection image including a reference object located in a reference area and a target object located in a target area;

[0043] Step S2, determining a first value of a first pixel channel corresponding to a reference area in the target detection image;

[0044] Step S3, determining a second value of a first pixel channel corresponding to the target object in the target detection image;

[0045] Step S4: determining the shielding condition of the target object according to the numerical relationship between the first value and the second value.

[0046] In this way, by setting the reference area and the target area, and comparing and analyzing the difference in pixel values ​​of the reference area and the target area in the first pixel channel in the target detection image, it is possible to automatically and quickly detect whether the target object is obscured, with high judgment accuracy. This method reduces dependence on manual judgment, does not require the aid of other detection equipment, avoids measurement errors, and improves detection efficiency and accuracy. In addition, this method does not rely on a specific type of image or target object. As long as a reasonable reference area is set and attention is paid to the difference in pixel channel values, complex image processing algorithms can be avoided. The calculation is simple and the versatility is strong.

[0047] In one embodiment, the target detection image may further include a number of reference objects, and the reference area represents the spatial range occupied by the reference objects.

[0048] In one embodiment, the target detection image may include a plurality of targets, and the target area refers to a spatial range occupied by the targets.

[0049] The target detection image refers to an image that needs to be occlusion detected. It can be an initial detection image obtained by a camera, sensor or other image capture device. The initial detection image is processed to obtain the target detection image. For example, the initial detection image is subjected to preprocessing operations such as denoising, contrast enhancement, brightness adjustment, etc., and / or the initial detection image is subjected to image segmentation, feature extraction and other operations.

[0050] In one embodiment, the target detection image is generated after encoding and decoding the initial detection image based on the visual converter. In this embodiment, the initial detection image is an actual photograph, and the target detection image is a corresponding grayscale image.

[0051] In the above embodiment, the target detection image can also be understood as a depth image, in which the value of the first pixel channel can represent different depth information, that is, the distance information of each pixel point in the target detection image from the camera. The distance information corresponds to the brightness value of the grayscale image, that is, the closer the target object is to the camera, the brighter the target area in the target detection image; the farther the target object is from the camera, the darker the target area in the target detection image.

[0052] Based on this, the first value can be a first brightness value determined according to the distance between the pixel point in the corresponding reference area in the target detection image and the camera; the second value can be a second brightness value determined according to the distance between the pixel point in the corresponding target area in the target detection image and the camera.

[0053] In a specific embodiment, the target detection image is determined based on a shallow image feature vector and a deep image feature vector of an initial detection image.

[0054] In one embodiment, the target detection image may also be a thermal imaging image (or infrared image), which is an image formed by capturing infrared radiation emitted from the surface of an object and reflects the temperature distribution of the object.

[0055] A reference object refers to a specific object or subject in a target detection image that needs to be identified and analyzed. It is used for comparative analysis with the target object. Its position and shape are known and can be used as a benchmark to judge the state of the target object. A reference area refers to a partial area containing a reference object. It is used to describe the shape, position and size of the reference object in the target detection image, and can also be used to extract feature information (such as pixel value) of the reference object.

[0056] The target object refers to the main object that needs to be detected whether it is blocked or not. The target area refers to the area in the target detection image excluding the reference area. The state of the target object (such as whether it is blocked or not) is inferred by analyzing the feature information (such as pixel value) of the target area.

[0057] In one embodiment, when the target object is completely imaged, the value of the first pixel channel corresponding to the target area in the detection image is less than the value of the first pixel channel corresponding to the reference area in the detection image. In this embodiment, "complete imaging of the target object" indicates that the target object is not blocked by any obstacles and can be fully displayed in the image. In this state, all the details and contours of the target object can be captured without missing or deforming due to occlusion, which means that, in an ideal situation (i.e., when there is no occlusion), the value of the first pixel channel of the target area is less than the value of the first pixel channel of the reference area. In practical applications, if it is detected that the pixel value of the target area suddenly increases and exceeds the pixel value of the reference area, it can be inferred that the target object may be partially or completely blocked.

[0058] Understandably, an image can be composed of multiple color channels (such as RGB, i.e., red, green, and blue channels), and the pixel value of each channel represents the intensity or brightness of the image at that color, so the "first pixel channel" can refer to any one of the three channels, or to the grayscale channel corresponding to the grayscale image. The detection image can be a target detection image (such as a grayscale image or a thermal imaging image) or an initial detection image (such as a photograph), and there is no specific limitation on this.

[0059] For ease of understanding, for example, Figure 2 As shown, for three-dimensional freezers with cameras installed on the cabinet doors, most of them are equipped with angle sensors. When the cabinet door is detected to be opened to a certain angle, the freezer surface is photographed. The door opening operation must be accompanied by the presence of the human body. Most of the small area occlusions generated in this process are accompanied by parts of the human body, such as hands, arms, etc. The external obstructions captured by the camera will definitely be on the outside of the baffle of each shelf. In this example, the picture taken by the freezer is the initial detection image, which is processed to obtain the corresponding target detection image. The baffle in the target detection image is the reference object. There can be multiple reference objects, and the corresponding width area is the reference area. Other areas such as stored items or hands (obstructions) are target objects, and the area where they are located is the target area.

[0060] In order to simplify the processing efficiency of image data and reduce the dimension of data, the captured initial detection image can be converted into a target detection image with grayscale characteristics.

[0061] like Figure 3 As shown, in one embodiment, before the step S1 of "obtaining a target detection image" in the present invention, the image occlusion detection method further includes the following steps.

[0062] Step S01, obtaining an initial detection image, extracting and determining a shallow image feature vector and a deep image feature vector of the initial detection image;

[0063] Step S02: Fusing the shallow image feature vector and the deep image feature vector to determine a first fused feature vector, and determining a target detection image according to the first fused feature vector.

[0064] In this way, by fusing the shallow characteristic feature vector and the deep image feature vector of the initial detection image, it is possible to maintain the overall recognition ability of the object while improving the sensitivity to the detailed differences of the object, making the first fused characteristic vector more comprehensive and discriminative.

[0065] Shallow image feature vectors refer to vector representations composed of features extracted from shallow layers of the network (i.e., layers close to the input layer). These features are closer to the original pixel information of the initial detection image and contain more details and local information. Deep image feature vectors refer to vector representations composed of features extracted from deeper layers of the network (i.e., layers far from the input layer). These features are more abstract and contain the overall structure and high-level semantic information of the image.

[0066] In a specific embodiment, a shallow image feature vector and a deep image feature vector of an initial detection image are extracted based on a vision transformer (ViT) model. In this embodiment, the vision transformer model includes a Transformer model, which uses an encoder-decoder architecture to input the initial detection image into the encoder of the ViT model to perform encoding operations to convert it into an image feature vector, and then the decoder based on the ViT model performs decoding operations to convert it into a target detection image.

[0067] As shown in FIG. 4( a ), in a specific embodiment, the part of “extracting and determining the shallow image feature vector and the deep image feature vector of the initial detection image” in step S01 of the present invention may specifically include the following steps.

[0068] Step M011, segmenting the initial detection image to determine a number of sub-image blocks;

[0069] Step M012, determining corresponding first eigenvectors and second eigenvectors according to the plurality of sub-image blocks, wherein the first eigenvectors are used to determine content information of the sub-image blocks, and the second eigenvectors are used to determine spatial position information of the sub-image blocks in the initial detection image;

[0070] Step M013, fusing the first feature vector and the second feature vector to determine a shallow image feature vector corresponding to the initial detection image.

[0071] In this way, by combining the content information and spatial position information of the sub-image block, the objects, textures and their distribution in the image can be more accurately reflected, providing a richer and more effective feature representation for subsequent image processing.

[0072] In a specific embodiment, the part of "extracting and determining the shallow image feature vector and the deep image feature vector of the initial detection image" in step S01 of the present invention may specifically include the following steps.

[0073] Step S01 ′: based on a multi-stage feature extraction structure, determine a corresponding deep image feature vector according to the shallow image feature vector.

[0074] In this way, multi-stage feature extraction operations can enhance the model's ability to understand and express image content, and promote in-depth analysis and integration of image features.

[0075] Furthermore, as shown in FIG. 4( b ), in a specific embodiment, step S01 ′ of the present invention may specifically include the following steps.

[0076] Step M021, according to the shallow image feature vector, a multi-stage downsampling feature extraction operation is used to determine a corresponding number of third feature vectors;

[0077] Step M022: Perform a multi-stage upsampling feature extraction operation on the plurality of third feature vectors to determine corresponding deep image feature vectors.

[0078] In this way, through the feature extraction operation combining multi-stage downsampling and upsampling, the progressive abstraction and detail recovery of image features from shallow to deep layers can be effectively achieved, enhancing the expressiveness and accuracy of deep image feature vectors.

[0079] Multi-stage feature extraction refers to the gradual extraction of multiple levels of features from raw data through a series of continuous transformation and processing steps. These features range from simple to complex, from concrete to abstract. Each stage is further extracted and transformed based on the output of the previous stage, which helps to capture the inherent laws and structures of the data.

[0080] Upsampling refers to increasing the size of a signal or image by increasing the number of samples or improving the resolution of the sample image. Upsampling can be used for image methods and super-resolution reconstruction. Upsampling can improve the resolution and clarity of an image while maintaining important information about the image. Downsampling is the opposite of upsampling. It refers to reducing the size of a signal or image by reducing the number of samples or reducing the resolution of the sample image. Downsampling can be used for image compression and feature extraction. Downsampling can reduce the size of an image while retaining as much important information as possible.

[0081] like Figure 5 As shown, in a specific embodiment, the part of "fusing the shallow image feature vector and the deep image feature vector to determine a first fused feature vector" in step S02 of the present invention may specifically include the following steps.

[0082] Step S021, according to the shallow image feature vector, a multi-stage downsampling feature extraction operation is used to determine a corresponding number of third feature vectors;

[0083] Step S022, performing a multi-stage upsampling feature extraction operation on the plurality of third feature vectors to determine a plurality of corresponding fourth feature vectors;

[0084] Step S023, fusing the third eigenvector and the corresponding fourth eigenvector to determine a plurality of second fused eigenvectors;

[0085] Based on this, the part of "determining the target detection image according to the first fused feature vector" in step S02 may specifically include the following steps.

[0086] Step S024: Determine a target detection image by performing a decoding operation according to the plurality of second fused feature vectors.

[0087] In this way, hierarchical features from shallow to deep layers are extracted through multi-stage downsampling (gradually abstracted from fine-grained to coarse-grained), which not only retains the local detail information of the image, but also captures the global semantic structure; then the upsampling operation is used to gradually restore the deep features to a resolution similar to that of the shallow features, so that the deep features can guide the shallow features for more accurate semantic understanding, thereby generating a more comprehensive and powerful second fused feature vector, effectively improving the richness of the image feature representation.

[0088] In a specific embodiment, Figure 6 As shown, the initial detection image is defined as X, and its corresponding dimension is W×H, where H and W are the height and width of the initial detection image respectively. X is divided into multiple non-overlapping sub-image blocks (or tokens). Assume that the size of each sub-image block is P×P (two-dimensional image block), so the entire initial detection image is divided into N=(W / P)×(H / P) sub-image blocks, where the height and width of the sub-image block are both P.

[0089] Flattening each sub-image block means converting the multi-dimensional vector representation of the sub-image block into a one-dimensional form for subsequent processing. Each sub-image block is flattened into a vector representation, and its corresponding dimension is P×P×C, where C is the channel of the image; the flattened representation of the sub-image block is input into the embedding layer (or linear layer) for linear projection operation to determine the conversion into a feature vector with a fixed dimension (i.e., the first feature vector), which is the format required by the Transformer model. The second feature vector can be expressed as:

[0090]

[0091] in, N is the number of sub-image blocks.

[0092] After the first feature vector and the second feature vector are input into the visual transcoder model to perform a transcoding operation (transformer), a shallow image feature vector is obtained. The shallow image feature vector can be expressed as:

[0093] f ressamplg (T) = ressamples (f T ), f ressample (T)∈N p *D^

[0094] Among them, s represents the size ratio of the processed feature vector to the input initial detection image, D^ represents the dimension of the output feature, T is a stage in the multi-stage, and f T represents the feature vector of the sub-image block after a transformer, f ressample (T) represents the fusion of image feature vectors after multi-stage processing to determine the multi-scale information flow after feature extraction at different stages.

[0095] According to the shallow image feature vector, a multi-stage upsampling feature extraction operation is used to determine the corresponding deep image feature vector, and the deep image feature vector can be expressed as:

[0096]

[0097] Among them, f p Represented as N and position embedding D is the feature dimension of the sub-image block mapped into the feature vector.

[0098] The deep image feature vector is then reshaped and concatenated to restore it to a feature map of the same dimension as the input, and the feature map is decoded into a target detection image.

[0099] In order to quickly convert the initial detection image into a target detection image that meets the requirements, the visual converter model can be trained to obtain a trained visual converter model; the initial detection image to be processed is input into the trained visual conversion model to determine the corresponding target detection image. The training process of the visual converter model is described in detail below.

[0100] A training dataset is obtained, wherein the training dataset includes multiple metadata datasets, such as selecting MIX6 as the main training dataset, which is composed of multiple sub-datasets from different sources and contains about 1.4 million images. The sub-datasets include at least one of DIW (Depth in the Wild), ETH3D, Sintel, KITTI, NYU Depth V2, and TUM.

[0101] Among them, DIW refers to a depth image dataset containing various natural scenes and artificial scenes, which is used to train the model to understand various complex depth information; ETH3D is a dataset used to provide high-quality 3D reconstruction datasets, including indoor and outdoor scenes. High-quality depth information helps to improve the depth prediction ability of the model; Sintel dataset, including synthetic scenes and animation scenes, is used to train the performance of the model in synthetic environments; KITTI dataset, including depth information of urban driving scenes, provides rich outdoor depth data, which is helpful for the performance of the model in practical applications; NYU Depth V2 dataset, including RGB-D dataset of indoor scenes, is used to improve the depth estimation ability of the model in indoor environments; TUM (Technical University of Munich) contains RGB-D dataset of indoor scenes, providing diverse indoor scene depth data, which is used to enhance the robustness of the model.

[0102] The training sample images in the training data set are resized to an appropriate size, such as adjusting the long side of the image to 384 pixels, maintaining the aspect ratio of the image, and scaling the short side accordingly; random horizontal flipping and / or random scaling (for example, the scaling ratio is between 0.5-2.0 times) are used for data enhancement to improve the generalization ability of the model.

[0103] During the pre-training process, a sub-sample data set can be selected for pre-training for several cycles (for example, 60 cycles). These sub-sample data may contain data that is more representative or of higher quality in MIX6. Each cycle contains 72,000 steps and the batch size is 16. After the pre-training is completed, the full training data set (MIX6) is used for another 60 cycles of training to further improve the performance of the model.

[0104] When fine-tuning is required on a smaller data set (such as DIW, etc.), the predicted value can be aligned to each training sample using a robust alignment procedure, which means that for each training sample, several adjustment parameters (such as scaling factors and / or offsets) are calculated to adjust the model's predicted value closer to the true value of the sample. After applying the alignment procedure to the training set, the scaling value and / or offset value of each sample is obtained, the average of all scaling values ​​is calculated, and / or the average of all offset values ​​is calculated, and the average scaling value and / or average offset value is determined.

[0105] During the training process, multiple levels of the visual converter model are used to extract different levels of features of the sample image, the image features extracted from different levels are reassembled and fused, and the fused image features are input into other parts of the model for prediction to generate a predicted detection image (or predicted depth map), and the training sample includes an initial detection image and a corresponding target detection image (or target depth map); based on the depth information difference between the predicted detection image and the target detection image, a loss function is determined; based on the loss function, the back propagation algorithm and the optimization algorithm are iteratively used until the loss function converges to determine the model parameters.

[0106] The optimization algorithm includes at least one of stochastic gradient descent and Adam optimizer. Among them, Adam optimizer can perform multi-objective optimization, and the optimizer can automatically adjust the learning rate, such as adjusting the learning rate of the model backbone network to 1e-5 and adjusting the learning rate of the decoder in the model to 1e-4, so as to promote the matching of learning rates of different parts.

[0107] During the training process, the model's predictions can be adjusted using the previously calculated average scaling value and / or average offset value before being passed to the loss function for error calculation. This ensures that the loss function is evaluated with optimized and adjusted predictions, thereby guiding the model to better learn how to adjust its predictions to match the true values ​​during training.

[0108] Using the trained visual transformer model, the input initial detection image (photo) can be converted into the corresponding target detection image (grayscale image or thermal imaging image). Specifically, the initial detection image is embedded with feature conversion (including sub-image block and position embedding), and the converted features are input into the Transformer for processing. Finally, through feature fusion and upsampling, a feature map of the same dimension is output, and the feature map is decoded to determine the output corresponding target detection image.

[0109] After obtaining the target detection image with grayscale characteristics, the target detection image can be scanned through a scanning frame (i.e., a convolution window). The scanning can be performed from left to right and from top to bottom. The size of the scanning area can be determined according to the size of the scanning frame. The size of the scanning frame can be adjusted according to the actual application, such as 3*3 or 6*6.

[0110] In one embodiment, the first value is determined according to the maximum value of the first pixel channel pixel values ​​corresponding to a plurality of reference objects in the reference area in the target detection image.

[0111] The pixel value of the first pixel channel may refer only to the brightness value in the grayscale image (because the grayscale image itself has only one brightness channel), or it may refer to multiple corresponding brightness values ​​of the brightness channels in a multi-channel color image (such as the Y channel in the YUV color space), and there is no specific limitation on this.

[0112] For example, as shown in Figures 7(a) and 7(b), in a refrigeration device with multiple internal baffles (i.e., reference objects), based on the different distances between different baffles and the camera (Figure 7(a)), the baffles are presented in different grayscale levels in the grayscale image (Figure 7(b)). For each identified baffle area, the grayscale values ​​of all pixels inside it are calculated. This can be achieved by traversing each pixel in the area and reading its grayscale value, comparing the maximum grayscale values ​​of all baffle areas, and determining the maximum value (i.e., the first value). The baffle corresponding to the maximum value is the brightest baffle in the grayscale image, i.e., k.

[0113] like Figure 8 As shown, in one embodiment, step S3 of the present invention may specifically include the following steps.

[0114] Step S31, determining a convolution window according to the first value, wherein the convolution window is used to scan a target detection image;

[0115] Step S32: determining and extracting a second value of a first pixel channel corresponding to a first target area in the target detection image based on the convolution window.

[0116] In this way, the target detection image is scanned through the convolution window, the feature value of the first target area is extracted, and the accurate positioning and extraction of the local features of the image are achieved.

[0117] The convolution window, also known as the convolution kernel window, is the matrix that the model uses to perform convolution processing on the convolution layer. The convolution window can be understood as a filter that can slide on the input data (target detection image) and calculate the local data to gradually obtain the processed data matrix, which retains some of the features of the original data.

[0118] The size or dimension of the convolution window determines the size of the scanning area or scanning frame. For example, if the convolution window is 3*3, that is, a 3*3 two-dimensional matrix, the size of the first target area is also 3*3.

[0119] In one embodiment, the target detection image can be stored in the form of a matrix, wherein each element represents the brightness value of the corresponding pixel point. Based on this, the target area includes a first target area, the first value forms a first matrix, including a number of reference brightness matrix elements corresponding to a number of pixel points in the reference area; the second value forms a second matrix, including a number of target brightness matrix elements corresponding to a number of pixel points in the first target area.

[0120] like Fig. 9 As shown, in one embodiment, step S4 of the present invention may specifically include the following steps.

[0121] Step S41, calculating the difference between the corresponding matrix elements of the second matrix and the first matrix, and summing all the differences to obtain a first measurement result;

[0122] Step S42: determining the occlusion condition of the first target area according to the first measurement result.

[0123] In this way, the first measurement result can reflect the difference between the first target area in the state to be detected and the state without obstruction. The method is simple, easy to understand and easy to implement.

[0124] In a specific embodiment, step S42 may specifically include the following steps.

[0125] Step S421, determining whether the first measurement result is greater than 0;

[0126] If yes, jump to step S422A, determine whether the first target area of ​​the target detection image is blocked, and set all elements of the second matrix to 1;

[0127] If not, jump to step S422B, determine that there is no occlusion in the first target area of ​​the target detection image, and set all elements of the second matrix to 0.

[0128] In this way, by updating the second matrix corresponding to the first target area, it is helpful to determine the position and size of the area where the target detection image is blocked. The presentation form is simple and clear and can be determined through matrix elements.

[0129] For ease of understanding, for example, as shown in Figure 10(a), assume that the input target detection image is a 6*6 two-dimensional matrix A, and the convolution window is a 3*3 two-dimensional matrix (i.e., the first matrix). When the convolution window scans to the upper left 3*3 first target area of ​​the matrix A, the corresponding upper left 3*3 two-dimensional matrix is ​​the second matrix. The corresponding elements of the second matrix and the first matrix are subtracted and then summed, i.e., (50-40)+(50-40)+(50-40)+(50-40)+(50-40)+(50-40)=60, i.e., the first measurement result is 60, which is greater than 0, then there is occlusion in the first target area, and all elements of the corresponding second matrix are set to 1.

[0130] When the convolution window scans from left to right to the second target area of ​​the upper right 3*3 area of ​​matrix A, the corresponding upper right 3*3 two-dimensional matrix is ​​the second matrix. The corresponding elements of the second matrix and the first matrix are subtracted and then summed, that is, (20-40)+(20-40)+(20-40)+(20-40)+(20-40)+(20-40)=-180, that is, the second measurement result is -180, which is less than 0. There is no occlusion in the second target area, and all elements of the corresponding second matrix are set to 0.

[0131] Similarly, the third target area of ​​3*3 at the lower left and the fourth target area of ​​3*3 at the upper right of matrix A are scanned, and the third measurement result is 60 and the fourth measurement result is -180 respectively. Therefore, the 6*6 two-dimensional matrix A corresponding to the target detection image finally outputs the 2*2 two-dimensional matrix B. Based on each element in the two-dimensional matrix B, it can be determined whether the corresponding target area is blocked. Figure 10(b) shows a display image with an occluder in the target detection image, and Figure 10(c) shows a display image with no occluder in the target detection image.

[0132] The above-mentioned various implementation modes, embodiments or specific examples provided by the present invention can be combined with each other to finally form multiple better implementation modes.

[0133] For example, Fig.11 The flow chart of the image occlusion detection method is shown below. Fig.11 , summarizing the processing process of this preferred embodiment.

[0134] An initial detection image is obtained, image segmentation is performed on the initial detection image, a reference object and a corresponding reference area are determined, and the initial detection image is input into a visual converter model for conversion operation to obtain a target detection image.

[0135] Based on the target detection image, a first value of a first pixel channel corresponding to a reference area in the target detection image; based on the first value, a convolution window (such as 3*3 or 7*7) is determined; based on the convolution window, the target detection image is scanned from left to right and from top to bottom to determine the second value of the first pixel channel of the first target area and the corresponding target object; based on the first value and the second value, a first measurement result is determined, and it is judged whether the first measurement result is greater than 0, if so, it is determined that the first target area is blocked, and if not, it is determined that the first target area is not blocked.

[0136] like Fig.12 As shown, an embodiment of the present invention provides a refrigeration device 200.

[0137] The refrigeration device 200 may specifically be a computer device, and the computer device may be a terminal device or a server.

[0138] The refrigeration device 200 includes at least one processor. The image occlusion detection method provided by the present invention can be applied to the processor or implemented by the processor. The processor can be a central processing unit (CPU) 21.

[0139] The refrigeration device 200 includes a memory. The memory is used to store various types of data to support the operation of the refrigeration device 200. Examples of such data include: any computer program for operating on a computer device. The memory may be a read-only memory (ROM) 22, a random access memory (RAM) 23, or another storage part 28. The storage part 28 may be located inside the refrigeration device 200 or outside the refrigeration device 200.

[0140] In one embodiment, when the processor executes the computer program stored in the memory, the steps of the image occlusion detection method in any technical solution of the present invention are performed.

[0141] In one embodiment, the refrigeration device 200 includes a central processor 21, which can perform various appropriate actions and processes according to the program stored in the read-only memory 22 or the program loaded from the storage part 28 to the random access memory 23. Various programs and data required for system operation are also stored in the random access memory 23. The central processor 21, the read-only memory 22 and the random access memory 23 are connected to each other through a bus 24. An input / output interface (Input / Output interface, i.e., I / O interface) 25 is also connected to the bus 24.

[0142] The following components are connected to the input / output interface 25: an input section 26 including a keyboard, a mouse, etc.; an output section 27 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 28 including a hard disk, etc.; and a communication section 29 including a network interface card such as a LAN card, a modem, etc. The communication section 29 performs communication processing via a network such as the Internet. A drive 210 is also connected to the input / output interface 25 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that a computer program read therefrom is installed into the storage section 28 as needed.

[0143] One embodiment of the present invention provides a computer-readable storage medium.

[0144] In one embodiment, a computer readable storage medium stores a computer program executed by the processor mentioned above, or an image occlusion detection method in any of the technical solutions mentioned above.

[0145] When the processor executes the computer program, it can execute the description of the image occlusion detection method in any of the above technical solutions, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated.

[0146] The computer-readable storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0147] In summary, the present invention provides an image occlusion detection method, device and storage medium. By setting a reference area and a target area, and comparing and analyzing the pixel value difference between the reference area and the target area in the target detection image in the first pixel channel, it is possible to automatically and quickly detect whether the target object is occluded, with high judgment accuracy. This method reduces the dependence on manual judgment, does not require the aid of other detection equipment, avoids measurement errors, and improves detection efficiency and accuracy; in addition, this method does not rely on a specific type of image or target object. As long as a reasonable reference area is set and attention is paid to the difference in pixel channel values, complex image processing algorithms can be avoided, and the calculation is simple and versatile.

[0148] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0149] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image occlusion detection method, characterized in that: include: Acquire a target detection image, the target detection image including a reference object located in a reference area and a target object located in a target area; When the target object is completely imaged, the value of the first pixel channel corresponding to the target area in the detection image is less than the value of the first pixel channel corresponding to the reference area in the detection image; Determine a first value of a first pixel channel corresponding to a reference area in the target detection image; Determine a second value of a first pixel channel corresponding to the target object in the target detection image; The occlusion condition of the target object is determined according to the numerical relationship between the first value and the second value.

2. The image occlusion detection method according to claim 1, characterized in that: The target area includes a first target area, the first values ​​form a first matrix including a number of reference brightness matrix elements corresponding to a number of pixel points in the reference area; the second values ​​form a second matrix including a number of target brightness matrix elements corresponding to a number of pixel points in the first target area; The determining, according to the numerical relationship between the first value and the second value, the shielding condition of the target object includes: Calculating the difference between the corresponding matrix elements of the second matrix and the first matrix, and summing all the differences to obtain a first measurement result; An occlusion condition of the first target area is determined according to the first measurement result.

3. The image occlusion detection method according to claim 2, characterized in that: The determining, according to the first measurement result, an occlusion condition of the first target area includes: Determine whether the first measurement result is greater than 0; If yes, it is determined that the first target area of ​​the target detection image is blocked, and all elements of the second matrix are set to 1; If not, it is determined that there is no occlusion in the first target area of ​​the target detection image, and all elements of the second matrix are set to 0.

4. The image occlusion detection method according to claim 1, characterized in that: The determining of a first value of a first pixel channel corresponding to a reference area in the target detection image includes: The first value is determined according to a maximum value of first pixel channel pixel values ​​corresponding to a plurality of reference objects in a reference area in the target detection image.

5. The image occlusion detection method according to claim 1, characterized in that: The determining of the second value of the first pixel channel corresponding to the target object in the target detection image includes: Determine a convolution window according to the first value, wherein the convolution window is used to scan the target detection image; Based on the convolution window, a second value of a first pixel channel corresponding to a first target area in the target detection image is determined and extracted.

6. The image occlusion detection method according to claim 1, characterized in that: Before acquiring the target detection image, the method includes: Acquire an initial detection image, and extract and determine a shallow image feature vector and a deep image feature vector of the initial detection image; The shallow image feature vector and the deep image feature vector are fused to determine a first fused feature vector, and a target detection image is determined according to the first fused feature vector.

7. The image occlusion detection method according to claim 6, characterized in that: The fusing the shallow image feature vector and the deep image feature vector to determine a first fused feature vector includes: According to the shallow image feature vector, a multi-stage downsampling feature extraction operation is used to determine a corresponding number of third feature vectors; Performing a multi-stage upsampling feature extraction operation on the plurality of third feature vectors to determine a corresponding plurality of fourth feature vectors; fusing the third eigenvector and the corresponding fourth eigenvector to determine a plurality of second fused eigenvectors; The determining the target detection image according to the first fused feature vector includes: According to the plurality of second fused feature vectors, a decoding operation is performed to determine a target detection image.

8. The image occlusion detection method according to claim 6, characterized in that: The extracting and determining of the shallow image feature vector and the deep image feature vector of the initial detection image includes: Segmenting the initial detection image to determine a number of sub-image blocks; Determine corresponding first eigenvectors and second eigenvectors according to the plurality of sub-image blocks, wherein the first eigenvectors are used to determine content information of the sub-image blocks, and the second eigenvectors are used to determine spatial position information of the sub-image blocks in the initial detection image; The first feature vector and the second feature vector are fused to determine a shallow image feature vector corresponding to an initial detection image.

9. The image occlusion detection method according to claim 6, characterized in that: The extracting and determining of the shallow image feature vector and the deep image feature vector of the initial detection image includes: Based on a multi-stage feature extraction structure, a corresponding deep image feature vector is determined according to the shallow image feature vector.

10. The image occlusion detection method according to claim 9, characterized in that: The method based on the multi-stage feature extraction structure determines the corresponding deep image feature vector according to the shallow image feature vector of the initial detection image, including: According to the shallow image feature vector, a multi-stage downsampling feature extraction operation is used to determine a corresponding number of third feature vectors; A multi-stage upsampling feature extraction operation is performed on the plurality of third feature vectors to determine corresponding deep image feature vectors.

11. A refrigeration device, characterized in that: include: A memory for storing executable instructions; A processor is used to implement the steps of the image occlusion detection method according to any one of claims 1 to 10 when running the executable instructions stored in the memory.

12. A computer-readable storage medium, comprising: at least one processor; A memory storing a computer program executable on the processor, wherein the processor executes the steps of the image occlusion detection method according to claims 1 to 10 when executing the program.

Images