Image processing method and device, electronic equipment and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a first image and a second image of a target region; the target region has a plurality of objects arranged in a stacking form; the first image and the second image correspond to different image acquisition time points; determining first information and second information according to the first image and the second image; the first information represents the change of the outer contour of the plurality of objects in the first image and the second image; the second information represents the change of the internal texture of the plurality of objects in the first image and the second image; and determining whether the plurality of objects are moved according to the first information and the second information.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] In some business scenarios, it's necessary to determine whether an object has been moved. For example, supply chain finance is a significant area of ​​business innovation in the logistics sector. In supply chain finance, logistics companies collaborate and mutually trust each other to provide services to merchants within the supply chain. One crucial service involves merchants pledging their goods to banks to apply for loans, and logistics companies, leveraging their warehousing advantages, provide the storage space for this collateral. A critical task for logistics companies in this process is ensuring the safety of the goods, guaranteeing that their condition is not altered in any way or by anyone without permission, such as being moved out of the warehouse. Only by achieving this can they provide reliable services to both merchants and banks, ensuring the smooth operation of supply chain finance.

[0003] However, there is still no effective solution in the relevant technologies for how to intelligently determine whether an object has been moved. Summary of the Invention

[0004] To address the related technical problems, embodiments of this application provide an image processing method, apparatus, electronic device, and storage medium.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an image processing method, including:

[0007] Acquire a first image and a second image of the target area; the target area contains multiple objects placed in a stacked manner; the first image and the second image were acquired at different times;

[0008] Based on the first image and the second image, first information and second information are determined; the first information represents the changes in the outer contours of the plurality of objects in the first image and the second image; the second information represents the changes in the internal textures of the plurality of objects in the first image and the second image.

[0009] Based on the first and second information, determine whether the plurality of objects have been moved.

[0010] In the above scheme, determining the first information based on the first image and the second image includes:

[0011] The first image is binarized using the first model to obtain the third image; and the second image is binarized using the first model to obtain the fourth image; the first model is trained using a semantic segmentation algorithm; the pixel values ​​of the pixels corresponding to the plurality of objects in the third image and the fourth image are first values, and the pixel values ​​of the other pixels besides the pixels corresponding to the plurality of objects are second values;

[0012] The first information is determined by comparing the third image and the fourth image.

[0013] In the above scheme, determining the first information by comparing the third image and the fourth image includes:

[0014] Based on the third image and the fourth image, a plurality of first coefficients are determined; each first coefficient characterizes whether the pixel value of a pixel in the third image is the same as the pixel value in the fourth image.

[0015] The first information is determined using the plurality of first coefficients.

[0016] In the above scheme, determining the first information using the plurality of first coefficients includes:

[0017] Using the plurality of first coefficients, a second coefficient is determined; the second coefficient characterizes the degree of matching between the third image and the fourth image.

[0018] Determine whether the second coefficient is greater than the first threshold; if the second coefficient is greater than the first threshold, the first information indicates that the outer contours of the multiple objects in the first image and the second image have changed; or, if the second coefficient is less than or equal to the first threshold, the first information indicates that the outer contours of the multiple objects in the first image and the second image have not changed.

[0019] The method in the above scheme further includes:

[0020] A first probability, a second probability, a third probability, and a fourth probability are determined for the first model; the first probability represents the probability that the first model identifies an object in the input image as an object; the second probability represents the probability that the first model identifies an object in the input image as a non-object; the third probability represents the probability that the first model identifies a non-object in the input image as an object; and the fourth probability represents the probability that the first model identifies a non-object in the input image as a non-object.

[0021] The first threshold is determined using the first probability, the second probability, the third probability, and the fourth probability.

[0022] In the above scheme, determining the second information based on the first image and the second image includes:

[0023] The first image is binarized using the second model to obtain the fifth image; and the second image is binarized using the second model to obtain the sixth image; the second model is trained using an edge detection algorithm; the pixel values ​​of the pixels corresponding to the edges in the fifth image and the sixth image are the first values, and the pixel values ​​of the pixels not corresponding to the edges are the second values;

[0024] The second information is determined using at least the fifth and sixth images.

[0025] In the above scheme, determining the second information by utilizing at least the fifth image and the sixth image includes:

[0026] The first image is binarized using the first model to obtain the third image; and the second image is binarized using the first model to obtain the fourth image; the first model is trained using a semantic segmentation algorithm; the pixel values ​​of the pixels corresponding to the plurality of objects in the third image and the fourth image are first values, and the pixel values ​​of the other pixels besides the pixels corresponding to the plurality of objects are second values;

[0027] The second information is determined using the third, fourth, fifth, and sixth images.

[0028] In the above scheme, determining the second information using the third image, the fourth image, the fifth image, and the sixth image includes:

[0029] Multiply the third image and the fifth image in pairs to obtain the seventh image; and multiply the fourth image and the sixth image in pairs to obtain the eighth image;

[0030] The second information is determined by comparing the seventh image and the eighth image.

[0031] In the above scheme, determining the second information by comparing the seventh image and the eighth image includes:

[0032] The seventh image is divided into multiple first grids based on preset rules; and the eighth image is divided into multiple second grids based on the preset rules.

[0033] Based on the plurality of first grids and the plurality of second grids, a plurality of third coefficients are determined; each third coefficient represents the degree of matching between a first grid and a corresponding second grid.

[0034] The second information is determined using the plurality of third coefficients.

[0035] In the above scheme, determining the second information using the plurality of third coefficients includes:

[0036] Determine whether each third coefficient is greater than the second threshold;

[0037] If a third coefficient is greater than the second threshold, the second information indicates that the internal texture of the plurality of objects in the first image and the second image has changed; or, if each third coefficient is less than or equal to the second threshold, the second information indicates that the internal texture of the plurality of objects in the first image and the second image has not changed.

[0038] The method in the above scheme further includes:

[0039] Determine the fifth, sixth, seventh, and eighth probabilities of the second model; the fifth probability represents the probability that the second model identifies an edge in the input image as an edge; the sixth probability represents the probability that the second model identifies an edge in the input image as a non-edge; the seventh probability represents the probability that the second model identifies a non-edge in the input image as an edge; and the eighth probability represents the probability that the second model identifies a non-edge in the input image as a non-edge.

[0040] The second threshold is determined using the fifth, sixth, seventh, and eighth probabilities.

[0041] In the above scheme, determining whether the plurality of objects have been moved based on the first information and the second information includes:

[0042] If the first information indicates that the outer contours of the plurality of objects have changed in the first image and the second image, and / or if the second information indicates that the internal textures of the plurality of objects have changed in the first image and the second image, it is determined that at least one of the plurality of objects has been moved.

[0043] or,

[0044] If the first information indicates that the outer contours of the plurality of objects in the first image and the second image have not changed, and the second information indicates that the internal textures of the plurality of objects in the first image and the second image have not changed, then it is determined that the plurality of objects have not been moved.

[0045] The method in the above scheme further includes:

[0046] When it is determined that at least one of the plurality of objects has been moved, an alarm message is issued; wherein,

[0047] The second information is determined using multiple third coefficients; each third coefficient represents the degree of matching between a first grid and a corresponding second grid; the first image corresponds to multiple first grids; the second image corresponds to multiple second grids; when the second information represents a change in the internal texture of the multiple objects in the first and second images, the alarm information includes at least one grid identifier; each grid identifier corresponds to a third coefficient greater than a second threshold; the at least one grid identifier is used to locate the moved object.

[0048] In the above scheme, acquiring the first image and the second image of the target area includes:

[0049] Acquire the ninth and tenth images of the first region; the first region at least includes the target region; the ninth and tenth images were acquired at different times.

[0050] Based on the ninth and tenth images, at least one second region is determined in the first region; the second region contains multiple objects arranged in a stacked manner;

[0051] A target region is determined from the at least one second region, and the ninth and tenth images are cropped based on the target region to obtain the first image and the second image.

[0052] In the above scheme, determining at least one second region in the first region based on the ninth and tenth images includes:

[0053] Using the ninth image, the tenth image, and the third model, at least one second region is determined in the first region; the third model is trained using an object detection algorithm.

[0054] This application also provides an image processing apparatus, including:

[0055] A first processing unit is used to acquire a first image and a second image of a target area; the target area contains multiple objects placed in a stacked manner; the first image and the second image are acquired at different times;

[0056] The second processing unit is configured to determine first information and second information based on the first image and the second image; the first information represents the changes in the outer contours of the plurality of objects in the first image and the second image; and the second information represents the changes in the internal textures of the plurality of objects in the first image and the second image.

[0057] The third processing unit is used to determine whether the plurality of objects have been moved based on the first information and the second information.

[0058] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor.

[0059] When the processor runs the computer program, it executes the steps of any of the above methods.

[0060] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0061] The image processing method, apparatus, electronic device, and storage medium provided in this application embodiment acquire a first image and a second image of a target area; the target area contains multiple objects placed in a stacked manner; the first image and the second image are acquired at different times; based on the first image and the second image, first information and second information are determined; the first information characterizes the changes in the outer contours of the multiple objects in the first image and the second image; the second information characterizes the changes in the internal textures of the multiple objects in the first image and the second image; based on the first information and the second information, it is determined whether the multiple objects have been moved. The solution provided in this application targets a target area containing multiple objects arranged in a stacked manner. Based on images acquired at different times, it identifies changes in the position of the objects from both overall and local perspectives. In other words, it determines whether multiple objects have been moved based on changes in the outer contours of the objects in the images (i.e., overall) and changes in the internal textures of the objects in the images (i.e., local). In this way, the state of the objects can be monitored through computer vision technology (i.e., image processing of the target area), thereby intelligently determining whether the objects have been moved and avoiding the waste of human resources for manual inspection. Furthermore, compared with manual inspection, by identifying changes in the position of objects from both overall and local perspectives, it is possible to identify subtle object movement events that are difficult for the human eye to perceive, thus improving the accuracy of the judgment results on whether objects have been moved. Attached Figure Description

[0062] Figure 1 This is a schematic flowchart of the image processing method according to an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of the outline and texture information of goods (i.e., objects) in an application embodiment of this application;

[0064] Figure 3 This is a schematic diagram of cargo area detection as an application embodiment of this application;

[0065] Figure 4 This is a schematic diagram of cargo area segmentation in an application embodiment of this application;

[0066] Figure 5 This is a schematic diagram of cargo texture detection as an application embodiment of this application;

[0067] Figure 6 This is a schematic diagram of the structure of the image processing apparatus according to an embodiment of this application;

[0068] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0069] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0070] In related technologies, manual inspection is typically used to determine whether an object has been moved, such as manually monitoring goods in a warehouse. This method wastes human resources, and the human eye cannot detect subtle movements of objects.

[0071] Based on this, in various embodiments of this application, for a target area containing multiple objects arranged in a stacked manner, changes in the position of the objects are identified from both overall and local perspectives based on images acquired at different times. In other words, the movement of multiple objects is determined based on changes in their outer contours (overall) and internal textures (local) in the images. Thus, the state of objects can be monitored using computer vision technology (i.e., image processing of the target area), thereby intelligently determining whether objects have been moved and avoiding the waste of human resources caused by manual inspection. Furthermore, compared with manual inspection, identifying changes in the position of objects from both overall and local perspectives can identify subtle object movement events that are difficult for the human eye to detect, improving the accuracy of the judgment result regarding whether objects have been moved.

[0072] This application provides an image processing method applied to electronic devices (such as servers), such as... Figure 1 As shown, the method includes:

[0073] Step 101: Obtain the first and second images of the target region;

[0074] Here, the target area contains multiple objects arranged in a stacked manner; the first image and the second image were acquired at different times;

[0075] Step 102: Determine the first information and the second information based on the first image and the second image;

[0076] Here, the first information represents the changes in the outer contours of the multiple objects in the first image and the second image; the second information represents the changes in the internal textures of the multiple objects in the first image and the second image.

[0077] Step 103: Determine whether the plurality of objects have been moved based on the first information and the second information.

[0078] Wherein, the first image corresponds to the first image acquisition time, and the second image corresponds to the second image acquisition time. It can be understood that determining whether the plurality of objects have been moved refers to judging whether the plurality of objects have been moved within the time range from the first image acquisition time to the second image acquisition time.

[0079] In practical applications, since the multiple objects are arranged in a stacked manner, they can be considered as a whole. When an object located at the edge of the multiple objects is moved, the outer contours of the multiple objects in the first and second images change; when an object located inside the multiple objects is moved, the internal texture of the multiple objects in the first and second images changes. The internal texture can be understood as the outline of the objects located inside the multiple objects and information such as the pattern on the outer packaging.

[0080] In step 101, in practical applications, acquiring the first and second images of the target area may include: acquiring the first and second images of the target area acquired by the image acquisition device, wherein the position and image acquisition angle of the image acquisition device are fixed.

[0081] In practical applications, some business scenarios may involve multiple objects stacked in various locations, such as goods piled up in a warehouse. To improve image processing efficiency, an image acquisition device can capture images containing multiple stacks of objects. The electronic device can acquire images containing multiple stacks of objects from the image acquisition device and detect the area where each stack of objects is located from the image. For each detected area, steps 101 to 103 are executed to determine whether any moved objects exist in each stack.

[0082] Based on this, in one embodiment, acquiring the first image and the second image of the target region may include:

[0083] Acquire the ninth and tenth images of the first region; the first region at least includes the target region; the ninth and tenth images were acquired at different times.

[0084] Based on the ninth and tenth images, at least one second region is determined in the first region; the second region contains multiple objects arranged in a stacked manner;

[0085] A target region is determined from the at least one second region, and the ninth and tenth images are cropped based on the target region to obtain the first image and the second image.

[0086] Here, it can be understood that the first area contains at least one stack of multiple objects placed in a stacked manner, and each second area contains one stack of multiple objects placed in a stacked manner.

[0087] In practical applications, the electronic device can acquire the ninth and tenth images captured by the image acquisition device. The ninth image corresponds to the acquisition time of the first image, and the tenth image corresponds to the acquisition time of the second image. Furthermore, when acquiring the ninth and tenth images, the position of the image acquisition device can be fixed or not, and the image acquisition angle of the image acquisition device can also be fixed or not. These settings can be customized according to requirements, and this embodiment does not impose any limitations on them.

[0088] In practical applications, it can be understood that when the position of the image acquisition device and / or the image acquisition angle are not fixed, it is necessary to process the ninth and tenth images through methods such as image comparison so that the objects in the ninth image can correspond to the objects in the tenth image.

[0089] In practical applications, a pre-trained model can be used to determine each group of objects in the first region, that is, to determine at least one second region in the first region.

[0090] Based on this, in one embodiment, determining at least one second region in the first region according to the ninth and tenth images may include:

[0091] Using the ninth image, the tenth image, and the third model, at least one second region is determined in the first region; the third model is trained using an object detection algorithm.

[0092] In practical applications, the target detection algorithm may include YOLOv5, Faster R-CNN, CenterNet, etc., and can be set according to requirements. This application embodiment does not limit this.

[0093] In practical applications, the third model needs to be pre-trained. Specifically, a training dataset can be determined, which may contain a predetermined number (e.g., 2000) images of a preset region (which can be set according to requirements, and this region needs to contain multiple objects placed in a stacked manner) acquired by the image acquisition device. Each stack of objects in each image is bounded, and the coordinate information corresponding to each stack of objects is recorded (i.e., labeled). After the labeling is completed, the labeled data and the object detection algorithm are used to train the third model.

[0094] In practical applications, it can be understood that when the position and / or image acquisition angle of the image acquisition device are not fixed, the ninth image needs to be input into the third model to obtain at least one candidate second region output by the third model, and then the tenth image needs to be input into the third model to obtain at least one candidate second region output by the third model. By associating the at least one candidate second region output by the third model based on the ninth image with the at least one candidate second region output by the third model based on the tenth image, at least one second region is determined (the candidate second region corresponding to the same pile of objects output by the third model in both instances can be determined as the second region).

[0095] With the position and image acquisition angle of the image acquisition device fixed, it is only necessary to input the ninth image into the third model to obtain at least one second region output by the third model.

[0096] In practical applications, the second region can be rectangular. By using the third model to detect the rectangular region occupied by each group of objects from the image acquired by the image acquisition device, it can be ensured that the subsequent image processing process is not interfered with by external information unrelated to the objects.

[0097] In step 102, in practical applications, a pre-trained model can be used to process the first image and the second image to determine the first information.

[0098] Based on this, in one embodiment, determining the first information based on the first image and the second image may include:

[0099] The first image is binarized using the first model to obtain the third image; and the second image is binarized using the first model to obtain the fourth image; the first model is trained using a semantic segmentation algorithm; the pixel values ​​of the pixels corresponding to the plurality of objects in the third image and the fourth image are first values, and the pixel values ​​of other pixels (i.e., non-object corresponding pixels) are second values.

[0100] The first information is determined by comparing the third image and the fourth image.

[0101] In practical applications, the first value can be 1, and the second value can be 0.

[0102] In practical applications, the semantic segmentation algorithm may include deeplab_v3, U-net, etc., and can be set according to requirements. This application embodiment does not limit this.

[0103] In practical applications, the first model needs to be pre-trained. Specifically, a training dataset can be determined, which may contain images (which can be the same as the images used to train the third model) of a predetermined region (which can be set according to requirements and requires multiple objects to be placed in a stacked manner) acquired by the image acquisition device. The coordinate positions of the outer contours of each stack of objects in each image are marked. The first model is trained using the marked data and the semantic segmentation algorithm.

[0104] In one embodiment, determining the first information by comparing the third image and the fourth image may include:

[0105] Based on the third image and the fourth image, a plurality of first coefficients are determined; each first coefficient characterizes whether the pixel value of a pixel in the third image is the same as the pixel value in the fourth image.

[0106] The first information is determined using the plurality of first coefficients.

[0107] In practical applications, the specific method for determining the first coefficient can be set according to requirements. For example, when the pixel value of a pixel in the third image is the same as the pixel value in the fourth image, the first coefficient can be equal to 0; when the pixel value of a pixel in the third image is different from the pixel value in the fourth image, the first coefficient can be equal to 1.

[0108] In one embodiment, determining the first information using the plurality of first coefficients may include:

[0109] Using the plurality of first coefficients, a second coefficient is determined; the second coefficient characterizes the degree of matching between the third image and the fourth image.

[0110] Determine whether the second coefficient is greater than the first threshold; if the second coefficient is greater than the first threshold, the first information indicates that the outer contours of the multiple objects in the first image and the second image have changed; or, if the second coefficient is less than or equal to the first threshold, the first information indicates that the outer contours of the multiple objects in the first image and the second image have not changed.

[0111] In practical applications, the specific method for calculating the second coefficient can be set according to requirements. It can be understood that the larger the second coefficient, the lower the matching degree between the third image and the fourth image.

[0112] In practical applications, the first threshold can be determined by statistically analyzing the performance of the first model on a preset validation dataset.

[0113] Based on this, in one embodiment, the method may further include:

[0114] A first probability, a second probability, a third probability, and a fourth probability are determined for the first model. The first probability represents the probability that the first model identifies an object in the input image (e.g., the first image and the second image) as an object, that is, the probability that the pixel value of the pixel corresponding to the object is determined as a first value during the binarization process of the input image. The second probability represents the probability that the first model identifies an object in the input image as a non-object, that is, the probability that the pixel value of the pixel corresponding to the object is determined as a second value during the binarization process of the input image. The third probability represents the probability that the first model identifies a non-object in the input image as an object, that is, the probability that the pixel value of the pixel corresponding to the non-object is determined as a first value during the binarization process of the input image. The fourth probability represents the probability that the first model identifies a non-object in the input image as a non-object, that is, the probability that the pixel value of the pixel corresponding to the non-object is determined as a second value during the binarization process of the input image.

[0115] The first threshold is determined using the first probability, the second probability, the third probability, and the fourth probability.

[0116] Here, the first probability, second probability, third probability, and fourth probability can be determined by statistically analyzing the performance of the first model on a preset validation dataset.

[0117] In practical applications, the specific method for determining the first threshold using the first, second, third, and fourth probabilities can be set according to requirements. For example, the first threshold can be determined by combining the first, second, third, and fourth probabilities with ideas such as Bernoulli distribution, binomial distribution, central limit theorem, Gaussian distribution, and the 3σ principle.

[0118] In step 102, in practical applications, a pre-trained model can be used to process the first and second images to determine the second information.

[0119] Based on this, in one embodiment, determining the second information based on the first image and the second image may include:

[0120] The first image is binarized using the second model to obtain the fifth image; and the second image is binarized using the second model to obtain the sixth image; the second model is trained using an edge detection algorithm; the pixel values ​​of the pixels corresponding to the edges in the fifth image and the sixth image are the first values, and the pixel values ​​of the pixels not corresponding to the edges are the second values;

[0121] The second information is determined using at least the fifth and sixth images.

[0122] Here, the first value can be 1, and the second value can be 0.

[0123] In practical applications, the edge detection algorithm may include PiDiNet, etc., and can be set according to requirements. This application embodiment does not limit this.

[0124] In practical applications, the second model can be pre-trained, or the second model can be an open-source model.

[0125] In practical applications, in order to further ensure that the subsequent image processing flow is not interfered with by external information unrelated to the object, the second information can be determined using the third image, the fourth image, the fifth image, and the sixth image.

[0126] Based on this, in one embodiment, determining the second information using at least the fifth image and the sixth image may include:

[0127] The first image is binarized using the first model to obtain the third image; and the second image is binarized using the first model to obtain the fourth image; the first model is trained using a semantic segmentation algorithm; the pixel values ​​of the pixels corresponding to the plurality of objects in the third image and the fourth image are first values, and the pixel values ​​of the other pixels besides the pixels corresponding to the plurality of objects are second values;

[0128] The second information is determined using the third, fourth, fifth, and sixth images.

[0129] Specifically, in one embodiment, determining the second information using the third image, the fourth image, the fifth image, and the sixth image may include:

[0130] Multiply the third image and the fifth image in pairs to obtain the seventh image; and multiply the fourth image and the sixth image in pairs to obtain the eighth image;

[0131] The second information is determined by comparing the seventh image and the eighth image.

[0132] Here, multiplying the third and fifth images in pairs, and multiplying the fourth and sixth images in pairs, can eliminate interference from external information unrelated to the object. In other words, the seventh and eighth images do not contain external information unrelated to the object. By comparing the seventh and eighth images, the second information is determined, ensuring that subsequent image processing is not interfered with by external information unrelated to the object, and further improving the accuracy of the judgment result.

[0133] In practical applications, there may be a large number of objects placed in a stacked manner. In order to further improve the accuracy of the judgment results, the seventh image and the eighth image can be divided into grids first, and then the grids of the seventh image and the eighth image can be compared in units of grids to determine the local texture changes in the multiple objects placed in a stacked manner.

[0134] Based on this, in one embodiment, determining the second information by comparing the seventh image and the eighth image includes:

[0135] The seventh image is divided into multiple first grids based on preset rules; and the eighth image is divided into multiple second grids based on the preset rules.

[0136] Based on the plurality of first grids and the plurality of second grids, a plurality of third coefficients are determined; each third coefficient represents the degree of matching between a first grid and a corresponding second grid.

[0137] The second information is determined using the plurality of third coefficients.

[0138] Here, the preset rules can be set according to requirements. For example, the preset rules may include: dividing the image into H×W grids, where H and W are both integers greater than 0, H and W can be the same or different, and the specific values ​​of H and W can be set based on experience.

[0139] Specifically, in one embodiment, determining the second information using the plurality of third coefficients may include:

[0140] Determine whether each third coefficient is greater than the second threshold;

[0141] If a third coefficient is greater than the second threshold, the second information indicates that the internal texture of the plurality of objects in the first image and the second image has changed; or, if each third coefficient is less than or equal to the second threshold, the second information indicates that the internal texture of the plurality of objects in the first image and the second image has not changed.

[0142] In practical applications, the second threshold can be determined by statistically analyzing the performance of the second model on a preset validation dataset.

[0143] Based on this, in one embodiment, the method may further include:

[0144] The fifth, sixth, seventh, and eighth probabilities of the second model are determined. The fifth probability represents the probability that the second model identifies an edge in the input image (e.g., the first image and the second image) as an edge, that is, the probability that the pixel value of the pixel corresponding to the edge is determined as a first value during the binarization process of the input image. The sixth probability represents the probability that the second model identifies an edge in the input image as a non-edge, that is, the probability that the pixel value of the pixel corresponding to the edge is determined as a second value during the binarization process of the input image. The seventh probability represents the probability that the second model identifies a non-edge in the input image as an edge, that is, the probability that the pixel value of the pixel corresponding to the non-edge is determined as a first value during the binarization process of the input image. The eighth probability represents the probability that the second model identifies a non-edge in the input image as a non-edge, that is, the probability that the pixel value of the pixel corresponding to the non-edge is determined as a second value during the binarization process of the input image.

[0145] The second threshold is determined using the fifth, sixth, seventh, and eighth probabilities.

[0146] Here, the fifth, sixth, seventh, and eighth probabilities can be determined by statistically analyzing the performance of the second model on a preset validation dataset.

[0147] In practical applications, the specific method for determining the second threshold using the fifth, sixth, seventh, and eighth probabilities can be set according to requirements. For example, the second threshold can be determined by combining the fifth, sixth, seventh, and eighth probabilities with concepts such as Bernoulli distribution, binomial distribution, central limit theorem, Gaussian distribution, and the 3σ principle.

[0148] For step 103, in one embodiment, determining whether the plurality of objects have been moved based on the first information and the second information may include:

[0149] If the first information indicates that the outer contours of the plurality of objects have changed in the first image and the second image, and / or if the second information indicates that the internal textures of the plurality of objects have changed in the first image and the second image, it is determined that at least one of the plurality of objects has been moved.

[0150] or,

[0151] If the first information indicates that the outer contours of the plurality of objects in the first image and the second image have not changed, and the second information indicates that the internal textures of the plurality of objects in the first image and the second image have not changed, then it is determined that the plurality of objects have not been moved.

[0152] In practical applications, when it is determined that at least one of the multiple objects has been moved, an alarm message can be sent to the target device to alert the user that there is a moved object in the target area.

[0153] Based on this, in one embodiment, the method may further include:

[0154] When it is determined that at least one of the plurality of objects has been moved, an alarm message is issued; wherein,

[0155] The second information is determined using multiple third coefficients; each third coefficient represents the degree of matching between a first grid and a corresponding second grid; the first image corresponds to multiple first grids; the second image corresponds to multiple second grids; when the second information represents a change in the internal texture of the multiple objects in the first and second images, the alarm information includes at least one grid identifier; each grid identifier corresponds to a third coefficient greater than a second threshold; the at least one grid identifier is used to locate the moved object.

[0156] In practical applications, the specific recipient of the alarm information (i.e., the target device) can be set according to requirements, and this application embodiment does not limit this.

[0157] In practical applications, the image processing method provided in this application can monitor whether a moved object exists in a specified area (e.g., a first area). The ninth image can serve as the original state image, reflecting the object's original state (e.g., its state upon entry into the warehouse); the tenth image can serve as the current state image, reflecting the object's current state. Furthermore, the ninth image can be updated based on the business status corresponding to the multiple objects (e.g., newly entered or exited goods). The tenth image can be updated periodically or triggered. Periodic updates may include the image acquisition device acquiring the tenth image of the first area at a preset period (which can be set as needed, e.g., n seconds, where n is a positive integer) and sending it to the electronic device; triggered updates may include the electronic device acquiring the tenth image from the image acquisition device upon receiving a detection command from another device (e.g., a terminal).

[0158] The image processing method provided in this application embodiment acquires a first image and a second image of a target region; the target region contains multiple objects placed in a stacked manner; the first image and the second image are acquired at different times; based on the first image and the second image, first information and second information are determined; the first information characterizes the changes in the outer contours of the multiple objects in the first image and the second image; the second information characterizes the changes in the internal textures of the multiple objects in the first image and the second image; based on the first information and the second information, it is determined whether the multiple objects have been moved. The solution provided in this application targets a target area containing multiple objects arranged in a stacked manner. Based on images acquired at different times, it identifies changes in the position of the objects from both overall and local perspectives. In other words, it determines whether multiple objects have been moved based on changes in the outer contours of the objects in the images (i.e., overall) and changes in the internal textures of the objects in the images (i.e., local). In this way, the state of the objects can be monitored through computer vision technology (i.e., image processing of the target area), thereby intelligently determining whether the objects have been moved and avoiding the waste of human resources for manual inspection. Furthermore, compared with manual inspection, by identifying changes in the position of objects from both overall and local perspectives, it is possible to identify subtle object movement events that are difficult for the human eye to perceive, thus improving the accuracy of the judgment results on whether objects have been moved.

[0159] The present application will be further described in detail below with reference to application examples.

[0160] This application example provides a computer vision-based warehouse goods (i.e., the aforementioned objects) monitoring solution, identifying changes in goods from both overall and local perspectives. The overall change in goods refers to changes in their outer contour, while the local change refers to changes in the texture of the corresponding area of ​​the goods in the image. Goods are typically stored in a stacked manner in a warehouse. If goods at the edge of a goods area in the image are moved, it will cause a change in the contour information of the corresponding location (i.e., the aforementioned first information), such as... Figure 2 As shown in box 201 in the image. If the goods within the goods area in the image are moved, it will cause a change in the texture information at the corresponding location (i.e., the second information mentioned above), the texture information is as follows. Figure 2 As shown in box 202.

[0161] In this application embodiment, in order to effectively identify the above phenomena (i.e., changes in contour information and changes in texture information), the warehouse goods management solution includes the following steps:

[0162] Step 1: Use the area detection model (i.e., the third model mentioned above) to detect cargo areas;

[0163] Step 2: Use the segmentation model (i.e., the first model mentioned above) to segment the cargo area;

[0164] Step 3: Perform cargo texture detection using the edge detection model (i.e., the second model mentioned above);

[0165] Step 4: Final Match.

[0166] The four steps described above complement each other. The specific implementation of each step is explained below.

[0167] First, the specific implementation of cargo area detection using the area detection model (i.e., the third model mentioned above) will be explained.

[0168] In this application embodiment, the function of the area detection model is to detect the rectangular area (i.e., the second area) occupied by the goods from the images (such as the ninth and tenth images mentioned above) acquired by the surveillance camera (i.e., the image acquisition device mentioned above), to ensure that the subsequent process is not interfered with by external information unrelated to the goods. The specific effects are as follows: Figure 3 As shown, each pile of goods will be framed to obtain a corresponding rectangular frame (e.g., rectangle 301).

[0169] In this application embodiment, the region detection model utilizes the YOLOv5 algorithm for region detection. During model training, approximately 2000 images detected by cameras in the warehouse are selected. For each image, each pile of goods is bounded, and the coordinates of each rectangle are recorded. After annotation, the labeled data is used to train the region detection model. The resulting region detection model can detect each pile of goods in newly acquired images and label the corresponding rectangles.

[0170] Secondly, the specific implementation of cargo area segmentation using the segmentation model (i.e., the first model mentioned above) will be explained.

[0171] In this application embodiment, the segmentation model's role is to obtain the outer contour of each pile of goods to determine whether the contour information of the goods has changed. The segmentation model uses the deeplab_v3 algorithm for image processing, and the output of the segmentation model (such as...) Figure 4 The image shown is a matrix of the same size as the input image (such as the third and fourth images mentioned above). The values ​​corresponding to the pixel positions of the goods in the matrix are 1 (i.e., the first value mentioned above), and the values ​​of the rest (i.e., the non-goods parts) are 0 (i.e., the second value mentioned above).

[0172] In this application embodiment, it is necessary to pre-acquire labeled data and use the labeled data to train the segmentation model. Acquiring labeled data can include selecting approximately 2000 images detected by cameras in the warehouse (images previously used in training the region detection model can be used), marking the coordinates of the outer contours of each pile of goods in the images, and training the segmentation model based on these markings. After training, the segmentation model analyzes new input images and generates a 0 / 1 matrix of the same size as the input image, where the region corresponding to the goods is 1, and the rest is 0.

[0173] Third, the specific implementation of cargo texture detection using the edge detection model (i.e., the second model mentioned above) is explained.

[0174] In this application embodiment, the edge detection model identifies local textures in the input image (such as the first and second images mentioned above) to determine whether the texture information has changed compared to the original. The edge detection model uses PiDiNet, which extracts important textures by identifying regions within the image where abrupt changes occur. The output image of the edge detection model is shown below. Figure 5 As shown, it is a 0, 1 matrix of the same size as the original image, with the pixel position corresponding to the edge having a value of 1 (i.e., the first value mentioned above), and the other positions having a value of 0 (i.e., the second value mentioned above).

[0175] In practical applications, since edge detection is a general algorithm and is not limited to the edge detection of goods, the edge detection model can be an open-source model and does not need to be retrained.

[0176] Fourth, the final matching process will be explained in detail.

[0177] In this application embodiment, the three steps of cargo region detection, cargo region segmentation, and cargo texture detection all serve the final recognition (i.e., final matching) process. After the goods enter the warehouse, a camera used to monitor the goods acquires an original image of the goods. Subsequently, subsequent images of the goods are periodically acquired. Each time an image is obtained, it needs to be compared with the original image to determine if the goods have moved. Both the original image and the subsequently acquired images of the goods undergo the three steps of cargo region detection, cargo region segmentation, and cargo texture detection before the final comparison.

[0178] In practical applications, the position and angle of surveillance cameras in warehouse scenarios are usually fixed. Therefore, it's possible to perform cargo region detection only on the original cargo image, and the resulting bounding boxes can also be used on subsequently acquired cargo images. However, both the original cargo image and the subsequently acquired cargo images need to undergo cargo region segmentation and cargo texture detection.

[0179] In practical applications, when the status of goods is updated, such as when new goods are received or shipped out, the original image can be updated.

[0180] In this application embodiment, after completing the above steps (i.e., cargo region detection, cargo region segmentation, and cargo texture detection), it is necessary to compare the original cargo image (represented as S, i.e., the ninth image above) and the subsequently obtained cargo image (represented as D, i.e., the tenth image above): first compare the contour information, and then compare the local information. The image may contain multiple piles of cargo, and each pile of cargo needs to be compared separately. Since the detection box (i.e., the second region) corresponding to each pile of cargo has been obtained in the cargo region detection process, the S and D images are cropped according to the detection box to obtain sub-images of each pile of cargo, which are represented as S(1), S(2), ... S(n) and D(1), D(2), ... D(n) respectively. Assuming that the sub-images S(i) (i.e., the first image above) and D(i) (i.e., the second image above) are to be compared, the process of comparing the contour information includes the following steps:

[0181] 1) Obtain the segmentation images corresponding to sub-images S(i) and D(i), denoted as Sg(i) (i.e. the third image above) and Dg(i) (i.e. the fourth image above).

[0182] 2) Calculate the coefficient of difference (i.e., the second coefficient mentioned above) using the following formula:

[0183]

[0184] Where (x,y) represents the coordinates of a position in the graph; Sg(i)(x,y) and Dg(i)(x,y) represent the values ​​of the corresponding positions in the two segmented graphs (i.e., Sg(i) and Dg(i)), respectively. This represents the XOR operation, which checks if the segmentation model agrees on whether a pixel in the original and later-acquired images represents a part of the goods. The output of the first coefficient (as mentioned above) is 0, otherwise it is 1. The larger the value of the difference coefficient, the less matched Sg(i) and Dg(i) are.

[0185] 3) If the difference coefficient is greater than a certain threshold (i.e. the first threshold mentioned above), it is determined that a pile of goods corresponding to the sub-graph has been moved.

[0186] The process of comparing texture information includes the following steps:

[0187] 1) Obtain the segmentation graphs corresponding to subgraphs S(i) and D(i), denoted as Sg(i) and Dg(i).

[0188] 2) Obtain the edge maps (output after using the edge detection model) corresponding to sub-images S(i) and D(i), denoted as Se(i) (i.e. the fifth image above) and De(i) (i.e. the sixth image above).

[0189] 3) Multiply Sg(i) and Se(i) in pairs to get St(i) (i.e., the seventh image above); and multiply Dg(i) and De(i) in pairs to get Dt(i) (i.e., the eighth image above).

[0190] 4) Divide St(i) and Dt(i) into H×W grids (i.e., the above preset strategy, where H and W are both integers greater than 0, and H and W can be the same or different) respectively. The part corresponding to each grid is represented as St(i)(h,w) and Dt(i)(h,w); H and W can be set according to experience.

[0191] 5) Calculate the difference coefficient (i.e., the third coefficient mentioned above) for each part of the grid using the following formula:

[0192]

[0193] in, and The meaning is similar, so I won't go into details here.

[0194] Furthermore, the reason for first dividing the image into a grid and then calculating the difference coefficient is that movement may only occur in a localized area. Directly comparing the entire image will yield relatively small differences, making it difficult to accurately determine whether the goods have moved. Grid division effectively solves this problem. In addition, grid division can also help pinpoint the specific location where the movement occurred.

[0195] 6) If the difference coefficient is greater than a certain threshold (i.e., the second threshold mentioned above), it is determined that the corresponding part (i.e., the region) in the corresponding grid has moved.

[0196] As can be seen from the above steps, comparing contours and textures requires comparing the difference coefficient with a corresponding threshold. Setting the threshold is often a challenging problem. Therefore, in this application embodiment, statistical methods are used to derive the threshold to help make more accurate judgments.

[0197] In this application embodiment, the derivation process of the threshold (i.e., the first threshold) when comparing contour information is as follows:

[0198] The probabilities of the segmentation model identifying a cargo portion as a cargo portion, a cargo portion as another portion, another portion as a cargo portion, and another portion as another portion are respectively denoted as p. TT (i.e., the first probability), p TF (i.e., the second probability), p FT (i.e., the third probability), p FF (i.e., the fourth probability), these probability values ​​can be obtained by evaluating the performance of the statistical segmentation model on the validation dataset. For Assuming its true value is represented as e(x,y), the value calculated by the model is If the two subgraphs are perfectly matched, then for any (x,y), e(x,y) = 0 is calculated as follows: The probability can be expressed as:

[0199]

[0200] Among them, the calculated The probability can be expressed as:

[0201]

[0202] Therefore, when e(x,y)=0, Follows Bernoulli distribution and Follows binomial distribution According to the central limit theorem, a Gaussian distribution can be used. Approximately, according to the 3σ principle, if two images are perfectly matched, The maximum value should not exceed Therefore, the threshold (first threshold) for the difference coefficient is set as follows:

[0203]

[0204] In this application embodiment, the derivation process of the threshold (i.e., the second threshold) when comparing texture information is as follows:

[0205] The probabilities of the edge detection model identifying an edge as an edge, an edge as a non-edge, a non-edge as an edge, and a non-edge as a non-edge are denoted as q. TT (i.e., the fifth probability), q TF (i.e., the sixth probability), q FT (i.e., the seventh probability), q FF (i.e., the eighth probability), these probability values ​​can be obtained by statistically analyzing the performance of edge detection models on validation datasets. For Assuming its true value is represented as g(x,y), the value calculated by the model is If the two grids are perfectly matched, then for any (x,y), g(x,y)=0, the calculated result is... The probability can be expressed as:

[0206]

[0207] Among them, the calculated The probability can be expressed as:

[0208]

[0209] Continuing with the same derivation process as the threshold for contour comparison (i.e., the first threshold), it can be deduced that the second threshold should be set as follows:

[0210]

[0211] In this application embodiment, the specific flow of the image processing procedure is as follows:

[0212] 1) Model Training. Data is labeled according to preset rules to train the region detection model and the segmentation model, and an edge detection model is obtained. Simultaneously, the probability values ​​p for the segmentation model and the edge detection model are estimated using the validation dataset. TT p TF p FT p FF and q TT q TF q FT q FF .

[0213] 2) Goods Area Inspection. After the goods have been received or their status updated, the corresponding cameras are used to capture the original images of the goods, and the inspection model is used to obtain the corresponding inspection boxes for each pile of goods.

[0214] 3) Cargo region segmentation. Cargo regions are segmented for the original cargo images to obtain the corresponding segmentation map. This operation only needs to be performed once if the cargo status is not updated. Cargo regions are segmented for periodically collected cargo images to obtain the corresponding segmentation map. This operation needs to be performed every time a new image is collected.

[0215] 4) Cargo texture detection. For the original cargo image, an edge detection algorithm is used to perform texture detection to obtain the corresponding edge image. If the cargo status is not updated, this operation only needs to be performed once. For periodically collected cargo images, edge detection is performed to obtain the corresponding edge image. This operation needs to be performed every time a new image is collected.

[0216] 5) Contour Comparison. Obtain the segmented sub-images Sg(i) and Dg(i) corresponding to each pair of sub-images S(i) and D(i), calculate their difference coefficient (i.e., the second coefficient), and derive the corresponding threshold (i.e., the first threshold). If the difference coefficient is higher than the threshold, it is determined that the corresponding goods have been moved, an alarm is triggered, and manual verification is requested.

[0217] 6) Texture Comparison. Obtain the mesh corresponding to each pair of sub-images S(i) and D(i), and the edge sub-images St(i)(h,w) and Dt(i)(h,w) corresponding to each mesh. Calculate their difference coefficient (i.e., the third coefficient) and derive the corresponding threshold (i.e., the second threshold). If the difference coefficient is higher than the threshold, it is determined that the corresponding area in the corresponding goods has been moved, an alarm is triggered, and manual verification is requested.

[0218] The solution provided in this application embodiment uses computer vision technology to monitor goods in the warehouse using surveillance cameras and determine whether they have been moved. Furthermore, it uses image segmentation and edge detection technology to obtain the outline and texture information of the goods, and compares the state (i.e., position) of the goods based on these two types of information to determine whether the goods have been moved from both overall and local dimensions. In addition, it uses statistical methods to derive a threshold for the difference between the original state and the post-collected state of the goods, which helps to make a more accurate judgment on whether the goods have been moved.

[0219] The solution provided in this application example uses computer vision technology to monitor the status (i.e., location) of goods in a warehouse. If a change in the status (i.e., a change in location) of the goods is detected, an alarm is triggered promptly, requesting manual verification. Since warehouses typically have a large number of surveillance cameras, this method can make full use of existing resources and effectively reduce manpower consumption. Furthermore, computer vision can also identify subtle changes that are difficult for the human eye to detect.

[0220] To implement the method of the embodiments of this application, the embodiments of this application also provide an image processing apparatus, which is disposed on an electronic device (e.g., installed on a server), such as... Figure 6 As shown, the device includes:

[0221] The first processing unit 601 is used to acquire a first image and a second image of a target area; the target area contains multiple objects placed in a stacked manner; the first image and the second image are acquired at different times;

[0222] The second processing unit 602 is configured to determine first information and second information based on the first image and the second image; the first information represents the changes in the outer contours of the plurality of objects in the first image and the second image; and the second information represents the changes in the internal textures of the plurality of objects in the first image and the second image.

[0223] The third processing unit 603 is used to determine whether the plurality of objects have been moved based on the first information and the second information.

[0224] In one embodiment, the second processing unit 602 is further configured to:

[0225] The first image is binarized using the first model to obtain the third image; and the second image is binarized using the first model to obtain the fourth image; the first model is trained using a semantic segmentation algorithm; the pixel values ​​of the pixels corresponding to the plurality of objects in the third image and the fourth image are first values, and the pixel values ​​of the other pixels besides the pixels corresponding to the plurality of objects are second values;

[0226] The first information is determined by comparing the third image and the fourth image.

[0227] In one embodiment, the second processing unit 602 is further configured to:

[0228] Based on the third image and the fourth image, a plurality of first coefficients are determined; each first coefficient characterizes whether the pixel value of a pixel in the third image is the same as the pixel value in the fourth image.

[0229] The first information is determined using the plurality of first coefficients.

[0230] In one embodiment, the second processing unit 602 is further configured to:

[0231] Using the plurality of first coefficients, a second coefficient is determined; the second coefficient characterizes the degree of matching between the third image and the fourth image.

[0232] Determine whether the second coefficient is greater than the first threshold; if the second coefficient is greater than the first threshold, the first information indicates that the outer contours of the multiple objects in the first image and the second image have changed; or, if the second coefficient is less than or equal to the first threshold, the first information indicates that the outer contours of the multiple objects in the first image and the second image have not changed.

[0233] In one embodiment, the second processing unit 602 is further configured to:

[0234] A first probability, a second probability, a third probability, and a fourth probability are determined for the first model; the first probability represents the probability that the first model identifies an object in the input image as an object; the second probability represents the probability that the first model identifies an object in the input image as a non-object; the third probability represents the probability that the first model identifies a non-object in the input image as an object; and the fourth probability represents the probability that the first model identifies a non-object in the input image as a non-object.

[0235] The first threshold is determined using the first probability, the second probability, the third probability, and the fourth probability.

[0236] In one embodiment, the second processing unit 602 is further configured to:

[0237] The first image is binarized using the second model to obtain the fifth image; and the second image is binarized using the second model to obtain the sixth image; the second model is trained using an edge detection algorithm; the pixel values ​​of the pixels corresponding to the edges in the fifth image and the sixth image are the first values, and the pixel values ​​of the pixels not corresponding to the edges are the second values;

[0238] The second information is determined using at least the fifth and sixth images.

[0239] In one embodiment, the second processing unit 602 is further configured to:

[0240] The first image is binarized using the first model to obtain the third image; and the second image is binarized using the first model to obtain the fourth image; the first model is trained using a semantic segmentation algorithm; the pixel values ​​of the pixels corresponding to the plurality of objects in the third image and the fourth image are first values, and the pixel values ​​of the other pixels besides the pixels corresponding to the plurality of objects are second values;

[0241] The second information is determined using the third, fourth, fifth, and sixth images.

[0242] In one embodiment, the second processing unit 602 is further configured to:

[0243] Multiply the third image and the fifth image in pairs to obtain the seventh image; and multiply the fourth image and the sixth image in pairs to obtain the eighth image;

[0244] The second information is determined by comparing the seventh image and the eighth image.

[0245] In one embodiment, the second processing unit 602 is further configured to:

[0246] The seventh image is divided into multiple first grids based on preset rules; and the eighth image is divided into multiple second grids based on the preset rules.

[0247] Based on the plurality of first grids and the plurality of second grids, a plurality of third coefficients are determined; each third coefficient represents the degree of matching between a first grid and a corresponding second grid.

[0248] The second information is determined using the plurality of third coefficients.

[0249] In one embodiment, the second processing unit 602 is further configured to:

[0250] Determine whether each third coefficient is greater than the second threshold;

[0251] If a third coefficient is greater than the second threshold, the second information indicates that the internal texture of the plurality of objects in the first image and the second image has changed; or, if each third coefficient is less than or equal to the second threshold, the second information indicates that the internal texture of the plurality of objects in the first image and the second image has not changed.

[0252] In one embodiment, the second processing unit 602 is further configured to:

[0253] Determine the fifth, sixth, seventh, and eighth probabilities of the second model; the fifth probability represents the probability that the second model identifies an edge in the input image as an edge; the sixth probability represents the probability that the second model identifies an edge in the input image as a non-edge; the seventh probability represents the probability that the second model identifies a non-edge in the input image as an edge; and the eighth probability represents the probability that the second model identifies a non-edge in the input image as a non-edge.

[0254] The second threshold is determined using the fifth, sixth, seventh, and eighth probabilities.

[0255] In one embodiment, the third processing unit 603 is further configured to:

[0256] If the first information indicates that the outer contours of the plurality of objects have changed in the first image and the second image, and / or if the second information indicates that the internal textures of the plurality of objects have changed in the first image and the second image, it is determined that at least one of the plurality of objects has been moved.

[0257] or,

[0258] If the first information indicates that the outer contours of the plurality of objects in the first image and the second image have not changed, and the second information indicates that the internal textures of the plurality of objects in the first image and the second image have not changed, then it is determined that the plurality of objects have not been moved.

[0259] In one embodiment, the device further includes a communication unit; the third processing unit 603 is further configured to issue an alarm message through the communication module when it determines that at least one of the plurality of objects has been moved; wherein,

[0260] The second information is determined using multiple third coefficients; each third coefficient represents the degree of matching between a first grid and a corresponding second grid; the first image corresponds to multiple first grids; the second image corresponds to multiple second grids; when the second information represents a change in the internal texture of the multiple objects in the first and second images, the alarm information includes at least one grid identifier; each grid identifier corresponds to a third coefficient greater than a second threshold; the at least one grid identifier is used to locate the moved object.

[0261] In one embodiment, the first processing unit 601 is further configured to:

[0262] Acquire the ninth and tenth images of the first region; the first region at least includes the target region; the ninth and tenth images were acquired at different times.

[0263] Based on the ninth and tenth images, at least one second region is determined in the first region; the second region contains multiple objects arranged in a stacked manner;

[0264] A target region is determined from the at least one second region, and the ninth and tenth images are cropped based on the target region to obtain the first image and the second image.

[0265] In one embodiment, the first processing unit 601 is further configured to determine at least one second region in the first region using the ninth image, the tenth image, and the third model; the third model is trained using an object detection algorithm.

[0266] In practical applications, the communication unit can be implemented by the communication interface in the image processing device; the first processing unit 601, the second processing unit 602 and the third processing unit 603 can be implemented by the processor in the image processing device.

[0267] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above-described program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the image processing apparatus and image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0268] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device, such as... Figure 7 As shown, the electronic device 700 includes:

[0269] The communication interface 701 enables information exchange with other electronic devices;

[0270] The processor 702 is connected to the communication interface 701 to enable information interaction with other electronic devices and to execute the methods provided by one or more of the above-mentioned technical solutions when running computer programs;

[0271] The memory 703 stores computer programs that can run on the processor 702.

[0272] Specifically, the processor 702 is used for:

[0273] Acquire a first image and a second image of the target area; the target area contains multiple objects placed in a stacked manner; the first image and the second image were acquired at different times;

[0274] Based on the first image and the second image, first information and second information are determined; the first information represents the changes in the outer contours of the plurality of objects in the first image and the second image; the second information represents the changes in the internal textures of the plurality of objects in the first image and the second image.

[0275] Based on the first and second information, determine whether the plurality of objects have been moved.

[0276] In one embodiment, the processor 702 is further configured to:

[0277] The first image is binarized using the first model to obtain the third image; and the second image is binarized using the first model to obtain the fourth image; the first model is trained using a semantic segmentation algorithm; the pixel values ​​of the pixels corresponding to the plurality of objects in the third image and the fourth image are first values, and the pixel values ​​of the other pixels besides the pixels corresponding to the plurality of objects are second values;

[0278] The first information is determined by comparing the third image and the fourth image.

[0279] In one embodiment, the processor 702 is further configured to:

[0280] Based on the third image and the fourth image, a plurality of first coefficients are determined; each first coefficient characterizes whether the pixel value of a pixel in the third image is the same as the pixel value in the fourth image.

[0281] The first information is determined using the plurality of first coefficients.

[0282] In one embodiment, the processor 702 is further configured to:

[0283] Using the plurality of first coefficients, a second coefficient is determined; the second coefficient characterizes the degree of matching between the third image and the fourth image.

[0284] Determine whether the second coefficient is greater than the first threshold; if the second coefficient is greater than the first threshold, the first information indicates that the outer contours of the multiple objects in the first image and the second image have changed; or, if the second coefficient is less than or equal to the first threshold, the first information indicates that the outer contours of the multiple objects in the first image and the second image have not changed.

[0285] In one embodiment, the processor 702 is further configured to:

[0286] A first probability, a second probability, a third probability, and a fourth probability are determined for the first model; the first probability represents the probability that the first model identifies an object in the input image as an object; the second probability represents the probability that the first model identifies an object in the input image as a non-object; the third probability represents the probability that the first model identifies a non-object in the input image as an object; and the fourth probability represents the probability that the first model identifies a non-object in the input image as a non-object.

[0287] The first threshold is determined using the first probability, the second probability, the third probability, and the fourth probability.

[0288] In one embodiment, the processor 702 is further configured to:

[0289] The first image is binarized using the second model to obtain the fifth image; and the second image is binarized using the second model to obtain the sixth image; the second model is trained using an edge detection algorithm; the pixel values ​​of the pixels corresponding to the edges in the fifth image and the sixth image are the first values, and the pixel values ​​of the pixels not corresponding to the edges are the second values;

[0290] The second information is determined using at least the fifth and sixth images.

[0291] In one embodiment, the processor 702 is further configured to:

[0292] The first image is binarized using the first model to obtain the third image; and the second image is binarized using the first model to obtain the fourth image; the first model is trained using a semantic segmentation algorithm; the pixel values ​​of the pixels corresponding to the plurality of objects in the third image and the fourth image are first values, and the pixel values ​​of the other pixels besides the pixels corresponding to the plurality of objects are second values;

[0293] The second information is determined using the third, fourth, fifth, and sixth images.

[0294] In one embodiment, the processor 702 is further configured to:

[0295] Multiply the third image and the fifth image in pairs to obtain the seventh image; and multiply the fourth image and the sixth image in pairs to obtain the eighth image;

[0296] The second information is determined by comparing the seventh image and the eighth image.

[0297] In one embodiment, the processor 702 is further configured to:

[0298] The seventh image is divided into multiple first grids based on preset rules; and the eighth image is divided into multiple second grids based on the preset rules.

[0299] Based on the plurality of first grids and the plurality of second grids, a plurality of third coefficients are determined; each third coefficient represents the degree of matching between a first grid and a corresponding second grid.

[0300] The second information is determined using the plurality of third coefficients.

[0301] In one embodiment, the processor 702 is further configured to:

[0302] Determine whether each third coefficient is greater than the second threshold;

[0303] If a third coefficient is greater than the second threshold, the second information indicates that the internal texture of the plurality of objects in the first image and the second image has changed; or, if each third coefficient is less than or equal to the second threshold, the second information indicates that the internal texture of the plurality of objects in the first image and the second image has not changed.

[0304] In one embodiment, the processor 702 is further configured to:

[0305] Determine the fifth, sixth, seventh, and eighth probabilities of the second model; the fifth probability represents the probability that the second model identifies an edge in the input image as an edge; the sixth probability represents the probability that the second model identifies an edge in the input image as a non-edge; the seventh probability represents the probability that the second model identifies a non-edge in the input image as an edge; and the eighth probability represents the probability that the second model identifies a non-edge in the input image as a non-edge.

[0306] The second threshold is determined using the fifth, sixth, seventh, and eighth probabilities.

[0307] In one embodiment, the processor 702 is further configured to:

[0308] If the first information indicates that the outer contours of the plurality of objects have changed in the first image and the second image, and / or if the second information indicates that the internal textures of the plurality of objects have changed in the first image and the second image, it is determined that at least one of the plurality of objects has been moved.

[0309] or,

[0310] If the first information indicates that the outer contours of the plurality of objects in the first image and the second image have not changed, and the second information indicates that the internal textures of the plurality of objects in the first image and the second image have not changed, then it is determined that the plurality of objects have not been moved.

[0311] In one embodiment, the processor 702 is further configured to issue an alarm message through the communication interface 701 when it determines that at least one of the plurality of objects has been moved; wherein,

[0312] The second information is determined using multiple third coefficients; each third coefficient represents the degree of matching between a first grid and a corresponding second grid; the first image corresponds to multiple first grids; the second image corresponds to multiple second grids; when the second information represents a change in the internal texture of the multiple objects in the first and second images, the alarm information includes at least one grid identifier; each grid identifier corresponds to a third coefficient greater than a second threshold; the at least one grid identifier is used to locate the moved object.

[0313] In one embodiment, the processor 702 is further configured to:

[0314] Acquire the ninth and tenth images of the first region; the first region at least includes the target region; the ninth and tenth images were acquired at different times.

[0315] Based on the ninth and tenth images, at least one second region is determined in the first region; the second region contains multiple objects arranged in a stacked manner;

[0316] A target region is determined from the at least one second region, and the ninth and tenth images are cropped based on the target region to obtain the first image and the second image.

[0317] In one embodiment, the processor 702 is further configured to determine at least one second region in the first region using the ninth image, the tenth image, and the third model; the third model is trained using an object detection algorithm.

[0318] It should be noted that the specific process by which the processor 702 performs the above operations is detailed in the method embodiment and will not be repeated here.

[0319] Of course, in practical applications, the various components in electronic device 700 are coupled together through bus system 704. It can be understood that bus system 704 is used to realize the connection and communication between these components. In addition to a data bus, bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 7The general designated all buses as Bus System 704.

[0320] The memory 703 in this embodiment is used to store various types of data to support the operation of the electronic device 700. Examples of such data include any computer program used to operate on the electronic device 700.

[0321] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 702. Processor 702 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the hardware of processor 702 or by instructions in software form. Processor 702 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 702 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically memory 703. Processor 702 reads information from memory 703 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0322] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0323] It is understood that the memory 703 in this embodiment can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0324] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 703 storing a computer program, which can be executed by the processor 702 of the electronic device 700 to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0325] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0326] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0327] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a first image and a second image of a target region; the target region has a plurality of objects arranged in a stacked manner; the first image and the second image correspond to different image acquisition time points; determining first information and second information according to the first image and the second image; the first information represents changes in the outer contours of the plurality of objects in the first image and the second image; the second information represents changes in the internal textures of the plurality of objects in the first image and the second image; determining whether the plurality of objects are moved according to the first information and the second information; wherein, according to the first image and the second image, the first information is determined by: using a first model to perform binaryzation processing on the first image to obtain a third image, and using the first model to perform binaryzation processing on the second image to obtain a fourth image; the first model is trained using a semantic segmentation algorithm; the pixel values of the pixel points corresponding to the plurality of objects in the third image and the fourth image are a first value, and the pixel values of other pixel points except the pixel points corresponding to the plurality of objects are a second value; determining the first information by comparing the third image and the fourth image; wherein, the first information is determined by comparing the third image and the fourth image, comprising: determining a plurality of first coefficients according to the third image and the fourth image; each first coefficient represents whether the pixel value of a pixel point in the third image is the same as the pixel value in the fourth image; determining the first information using the plurality of first coefficients; wherein, the first information is determined using the plurality of first coefficients, comprising: determining a second coefficient using the plurality of first coefficients; the second coefficient represents the matching degree of the third image and the fourth image; determining whether the second coefficient is greater than a first threshold value; in the case that the second coefficient is greater than the first threshold value, the first information represents that the outer contours of the plurality of objects in the first image and the second image have changed; or, in the case that the second coefficient is less than or equal to the first threshold value, the first information represents that the outer contours of the plurality of objects in the first image and the second image have not changed.

2. The method of claim 1, wherein, The method further comprises: determining a first probability, a second probability, a third probability and a fourth probability of the first model; the first probability represents the probability that the first model identifies an object in an input image as an object; the second probability represents the probability that the first model identifies an object in an input image as a non-object; the third probability represents the probability that the first model identifies a non-object in an input image as an object; the fourth probability represents the probability that the first model identifies a non-object in an input image as a non-object; determining the first threshold value using the first probability, the second probability, the third probability and the fourth probability.

3. The method of claim 1, wherein, The second information is determined according to the first image and the second image, comprising: binarize the first image by using a second model to obtain a fifth image, and binarize the second image by using the second model to obtain a sixth image, wherein the second model is trained by using an edge detection algorithm, and pixel values of pixel points corresponding to edges in the fifth image and the sixth image are a first value, and pixel values of pixel points not corresponding to the edges are a second value; determine the second information by using at least the fifth image and the sixth image.

4. The method of claim 3, wherein, The determining the second information by using at least the fifth image and the sixth image comprises: binarize the first image by using a first model to obtain a third image, and binarize the second image by using the first model to obtain a fourth image, wherein the first model is trained by using a semantic segmentation algorithm, and pixel values of pixel points corresponding to the plurality of objects in the third image and the fourth image are the first value, and pixel values of other pixel points except the pixel points corresponding to the plurality of objects are the second value; determine the second information by using the third image, the fourth image, the fifth image and the sixth image.

5. The method of claim 4, wherein, The determining the second information by using the third image, the fourth image, the fifth image and the sixth image comprises: multiply the third image and the fifth image to obtain a seventh image, and multiply the fourth image and the sixth image to obtain an eighth image; determine the second information by comparing the seventh image and the eighth image.

6. The method of claim 5, wherein, The determining the second information by comparing the seventh image and the eighth image comprises: divide the seventh image based on a preset rule to obtain a plurality of first grids, and divide the eighth image based on the preset rule to obtain a plurality of second grids; determine a plurality of third coefficients according to the plurality of first grids and the plurality of second grids, wherein each third coefficient represents a matching degree of a first grid and a corresponding second grid; determine the second information by using the plurality of third coefficients.

7. The method of claim 6, wherein, The determining the second information by using the plurality of third coefficients comprises: determine whether each third coefficient is greater than a second threshold value; in a case where there is a third coefficient greater than the second threshold value, the second information represents that internal textures of the plurality of objects in the first image and the second image change, or in a case where each third coefficient is less than or equal to the second threshold value, the second information represents that internal textures of the plurality of objects in the first image and the second image do not change.

8. The method of claim 7, wherein, The method further comprises: determine a fifth probability, a sixth probability, a seventh probability and an eighth probability of the second model, wherein the fifth probability represents a probability that the second model identifies an edge in an input image as an edge, the sixth probability represents a probability that the second model identifies the edge in the input image as a non-edge, the seventh probability represents a probability that the second model identifies a non-edge in the input image as the edge, and the eighth probability represents a probability that the second model identifies the non-edge in the input image as the non-edge. The second threshold is determined using the fifth probability, the sixth probability, the seventh probability, and the eighth probability.

9. The method according to any one of claims 1 to 8, characterized in that, The determining whether the plurality of objects are moved according to the first information and the second information comprises: In a case where the first information represents that the outer contours of the plurality of objects in the first image and the second image change, and / or in a case where the second information represents that the internal textures of the plurality of objects in the first image and the second image change, it is determined that at least one object of the plurality of objects is moved; Or, In a case where the first information represents that the outer contours of the plurality of objects in the first image and the second image do not change, and the second information represents that the internal textures of the plurality of objects in the first image and the second image do not change, it is determined that the plurality of objects are not moved.

10. The method of claim 9, wherein, The method further comprises: When it is determined that at least one object of the plurality of objects is moved, an alarm information is sent; wherein The second information is determined using a plurality of third coefficients; each third coefficient represents a matching degree of a first grid and a corresponding second grid; the first image corresponds to a plurality of first grids; the second image corresponds to a plurality of second grids; in a case where the second information represents that the internal textures of the plurality of objects in the first image and the second image change, the alarm information comprises at least one grid identifier; each grid identifier corresponds to a third coefficient greater than a second threshold; the at least one grid identifier is used to locate the moved object.

11. The method according to any one of claims 1 to 8, characterized in that, The obtaining of the first image and the second image of the target region comprises: Obtaining a ninth image and a tenth image of a first region; the first region at least contains the target region; the ninth image and the tenth image correspond to different image acquisition time points; Determining at least one second region in the first region according to the ninth image and the tenth image; the second region contains a plurality of objects placed in a stacking form; Determining the target region from the at least one second region, and cropping the ninth image and the tenth image based on the target region to obtain the first image and the second image.

12. The method of claim 11, wherein, The determining of the at least one second region in the first region according to the ninth image and the tenth image comprises: Determining at least one second region in the first region using the ninth image, the tenth image, and a third model; the third model is trained using a target detection algorithm.

13. An image processing apparatus characterized by comprising: It comprises: A first processing unit is configured to obtain a first image and a second image of a target region; The target region contains a plurality of objects placed in a stacking form; The first image and the second image correspond to different image acquisition time points; A second processing unit is configured to determine first information and second information according to the first image and the second image; The first information represents a change of outer contours of the plurality of objects in the first image and the second image; the second information represents a change of internal textures of the plurality of objects in the first image and the second image; wherein the determining of the first information according to the first image and the second image comprises: binarize the first image using the first model to obtain a third image, and binarize the second image using the first model to obtain a fourth image; the first model is trained using a semantic segmentation algorithm; pixel values of pixel points corresponding to the plurality of objects in the third image and the fourth image are a first value, and pixel values of other pixel points except the pixel points corresponding to the plurality of objects are a second value; determine the first information by comparing the third image and the fourth image; wherein the determining the first information by comparing the third image and the fourth image comprises: determine a plurality of first coefficients according to the third image and the fourth image; each first coefficient represents whether a pixel value of a pixel point in the third image is the same as a pixel value of the pixel point in the fourth image; determine the first information using the plurality of first coefficients; wherein the determining the first information using the plurality of first coefficients comprises: determine a second coefficient using the plurality of first coefficients; the second coefficient represents a matching degree of the third image and the fourth image; determine whether the second coefficient is greater than a first threshold value; in a case where the second coefficient is greater than the first threshold value, the first information represents that an outer contour of the plurality of objects in the first image and the second image changes; or, in a case where the second coefficient is less than or equal to the first threshold value, the first information represents that the outer contour of the plurality of objects in the first image and the second image does not change; a third processing unit configured to determine whether the plurality of objects are moved according to the first information and second information.

14. An electronic device, comprising: comprise: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor is configured to execute the steps of the method of any one of claims 1 to 12 when running the computer program.

15. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 12.

Images