Image processing method, device, electronic device, and storage medium

Abstract

The present application discloses an image processing method, device, electronic device and storage medium, which are related to the technical field of artificial intelligence, and includes the steps of: acquiring a first image and a second image of a target area where a plurality of objects are placed in a stacked manner, the first image and the second image are captured at different times; determining, based on the first image and the second image, first information representing changes in the outer contours of the plurality of objects in the first image and the second image and second information representing changes in the inner textures of the plurality of objects in the first image and the second image; and determining, based on the first information and the second information, whether the plurality of objects have been moved.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority from a Chinese patent application bearing application number 202210350472.4 and filed on April 2, 2022, the entire contents of which are hereby incorporated by reference into this application.

[0002] The present application relates to the technical field of artificial intelligence, and in particular to an image processing method, device, electronic device, and storage medium. [Background technology]

[0003] In some business scenarios, it is necessary to determine whether an object has been moved. For example, supply chain financing is currently an important business innovation direction in the logistics industry. In supply chain financing, logistics companies and banks cooperate to build mutual trust and provide services to supply chain businesses. One important service is allowing businesses to apply for loans by pledging their cargo with banks. Since logistics companies have advantages in warehousing, they provide a place for the cargo to be pledged. In this process, one of the logistics companies' key tasks is to ensure the security of the cargo, that is, to ensure that the cargo cannot be moved out of the warehouse without authorization or otherwise altered in any way by anyone. Only in this way can they provide reliable services to businesses and banks and ensure the smooth operation of supply chain financing businesses.

[0004] However, in the related art, there is still no effective solution for how to intelligently determine whether an object has been moved. Summary of the Invention [Problem to be solved by the invention]

[0005] To solve the problems in the related art, embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium. [Means for solving the problem]

[0006] The technical aspects of the embodiments of the present application are realized as follows.

[0007] The present embodiment is acquiring a first image and a second image of a target area where a plurality of objects are stacked, the first image and the second image being captured at different times; determining, based on the first and second images, first information representing a change in the outer contour of the plurality of objects in the first and second images and second information representing a change in the inner texture of the plurality of objects in the first and second images; determining whether the plurality of objects have been moved based on the first information and the second information.

[0008] In the above aspect, determining the first information based on the first image and the second image includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values, and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; and determining the first information by comparing the third image with the fourth image.

[0009] In the above aspect, determining the first information by comparing the third image with the fourth image includes: determining a plurality of first coefficients based on the third image and the fourth image, each coefficient representing whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; determining the first information using the plurality of first coefficients.

[0010] In the above aspect, determining the first information using the plurality of first coefficients includes: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; Determining whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have not changed.

[0011] In the above aspect, the method comprises: determining a first probability, a second probability, a third probability, and a fourth probability of the first model, wherein the first probability represents a probability that the first model recognizes an object in the input image as an object, the second probability represents a probability that the first model recognizes an object in the input image as a non-object, the third probability represents a probability that the first model recognizes a non-object in the input image as an object, and the fourth probability represents a probability that the first model recognizes a non-object in the input image as a non-object; The method further includes determining the first threshold using the first probability, the second probability, the third probability, and the fourth probability.

[0012] In the above aspect, determining the second information based on the first image and the second image includes: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image, binarizing the second image using the second model to obtain a sixth image, and setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; and determining the second information from at least the fifth image and the sixth image.

[0013] In the above aspect, determining the second information using at least the fifth image and the sixth image includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values, and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; determining the second information from the third image, the fourth image, the fifth image, and the sixth image.

[0014] In the above aspect, determining the second information using the third image, the fourth image, the fifth image, and the sixth image includes: aligning and multiplying the third image and the fifth image to obtain a seventh image, and aligning and multiplying the fourth image and the sixth image to obtain an eighth image; and determining the second information by comparing the seventh image with the eighth image.

[0015] In the above aspect, determining the second information by comparing the seventh image with the eighth image includes: Dividing the seventh image into a plurality of first grids based on a predetermined rule, and dividing the eighth image into a plurality of second grids based on the predetermined rule; determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a matching degree between one of the first lattices and a corresponding one of the second lattices; determining the second information using the plurality of third coefficients.

[0016] In the above aspect, determining the second information using the plurality of third coefficients includes: determining whether each third coefficient is greater than a second threshold; The second information includes: if there is a third coefficient greater than the second threshold, the second information represents that the internal texture in the first image and the second image of the plurality of objects has changed; or if each third coefficient is less than or equal to the second threshold, the second information represents that the internal texture in the first image and the second image of the plurality of objects has not changed.

[0017] In the above aspect, the method comprises: determining a fifth probability, a sixth probability, a seventh probability, and an eighth probability of the second model, wherein the fifth probability represents the probability that the second model recognizes an edge in the input image as an edge, the sixth probability represents the probability that the second model recognizes an edge in the input image as a non-edge, the seventh probability represents the probability that the second model recognizes a non-edge in the input image as an edge, and the eighth probability represents the probability that the second model recognizes a non-edge in the input image as a non-edge; The method further includes determining the second threshold using the fifth probability, the sixth probability, the seventh probability, and the eighth probability.

[0018] In the above aspect, determining whether the plurality of objects have been moved based on the first information and the second information includes: determining that at least one of the plurality of objects has been moved if the first information indicates that an outer contour of the plurality of objects in the first image and the second image has changed and / or if the second information indicates that an inner texture of the plurality of objects in the first image and the second image has changed; Alternatively, the method includes determining that the plurality of objects have not moved if the first information indicates that the outer contours of the plurality of objects in the first image and the second image have not changed, and the second information indicates that the internal textures of the plurality of objects in the first image and the second image have not changed.

[0019] In the above aspect, the method comprises: and issuing a warning when it is determined that at least one object among the plurality of objects has been moved; Wherein, the second information is determined by a plurality of third coefficients each representing the degree of matching between one first grid and a corresponding second grid, and when the first image corresponds to a plurality of first grids and the second image corresponds to a plurality of second grids, and the second information represents that the internal textures of the plurality of objects in the first image and the second image have changed, the warning information includes at least one grid marker for locating the moved object, each corresponding to a third coefficient greater than a second threshold.

[0020] In the above aspect, acquiring the first image and the second image of the target area includes: acquiring a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image corresponding to different image acquisition times; determining, based on the ninth and tenth images, at least one second area in which a plurality of objects placed in a stacked manner are present in the first area; determining a target area from the at least one second area; and cropping the ninth and tenth images based on the target area to obtain the first and second images.

[0021] In the above aspect, determining at least one second region in the first region based on the ninth image and the tenth image includes: determining at least one second region in the first region using the ninth image, the tenth image, and a third model trained with a target detection algorithm;

[0022] The present embodiment is a first processing unit configured to acquire a first image and a second image of a target area where a plurality of objects are present in a stacked manner, the first image and the second image having different corresponding image acquisition times; a second processing unit configured to determine, based on the first and second images, first information representing changes in outer contours of the plurality of objects in the first and second images and second information representing changes in inner textures of the plurality of objects in the first and second images; and a third processing unit configured to determine whether the plurality of objects have been moved based on the first information and the second information.

[0023] An embodiment of the present application comprises a processor and a memory adapted to store a computer program executable by the processor; The present invention further provides an electronic device that is used to perform the steps of any of the above methods when the processor runs the computer program.

[0024] An embodiment of the present application further provides a storage medium having stored thereon a computer program that, when executed by a processor, implements the steps of any of the above methods. [Effects of the Invention]

[0025] The image processing method, device, electronic device, and storage medium according to the embodiments of the present application acquire first and second images of a target area in which multiple objects placed in a stacked manner exist, with corresponding image capture times being different, and determine, based on the first and second images, first information representing changes in the outer contours of the multiple objects in the first and second images and second information representing changes in the internal textures of the multiple objects in the first and second images, and determine, based on the first and second information, whether the multiple objects have been moved. In an embodiment of the present application, for a target area where multiple objects are stacked, changes in object position are recognized from two perspectives, namely, global and local, based on images taken at different times. In other words, whether multiple objects have been moved is determined based on changes in the outer contours (i.e., global) in the images of the multiple objects and changes in the internal textures (i.e., local) in the images of the multiple objects. In this way, the status of the objects is monitored using computer vision technology (i.e., processing of images of the target area), and whether an object has been moved is intelligently determined, avoiding the waste of human resources caused by manual inspections. Furthermore, compared to manual inspections, by recognizing changes in object position from two perspectives, global and local, subtle object movement events that are difficult for the human eye to notice can be recognized, and the accuracy of the determination results of whether an object has been moved can be improved. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a schematic diagram illustrating the flow of an image processing method according to an embodiment of the present application. [Figure 2] 3 is a schematic diagram of contour information and texture information of cargo (i.e., object) according to an application example of the present application. [Figure 3] 1 is a schematic diagram of cargo area detection according to an application example of the present application. FIG. [Figure 4] 1 is a schematic diagram of cargo area division according to an application example of the present application. FIG. [Figure 5] 1 is a schematic diagram of cargo texture detection according to an application example of the present application; [Figure 6]1 is a structural schematic diagram of an image processing device according to an embodiment of the present application; [Figure 7] 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0027] The present invention will now be described in more detail with reference to the drawings and examples.

[0028] In the related art, whether an object has been moved is usually determined by human patrol inspection, for example, by human patrol inspection to monitor cargo in a warehouse, which wastes human resources and is difficult for the human eye to notice some subtle object movement events.

[0029] Based on this, in each embodiment of the present application, for a target area where multiple objects are placed in a stacked manner, changes in object position are recognized from two perspectives, namely, global and local, based on images taken at different times. In other words, it is determined whether multiple objects have been moved based on changes in the outer contours of the images of the multiple objects (i.e., global) and changes in the internal texture of the images of the multiple objects (i.e., local). In this way, the status of the objects is monitored using computer vision technology (i.e., processing of images of the target area), and intelligently determining whether an object has been moved can be achieved, avoiding the waste of human resources caused by manual inspections. Furthermore, compared to manual inspections, by recognizing changes in object position from two perspectives, global and local, it is possible to recognize subtle object movement events that are difficult for the human eye to notice, and improve the accuracy of the determination results of whether an object has been moved.

[0030] An embodiment of the present application provides an image processing method applied to an electronic device (e.g., a server), as shown in FIG. 1 , the method includes: In step 101, a first image and a second image of a target area are acquired; wherein there are a plurality of objects placed in a stacked manner in the target area, and the corresponding image acquisition times of the first image and the second image are different; In step 102, first information and second information are determined based on the first image and the second image; wherein the first information represents a change in outer contour of the plurality of objects in the first image and the second image, and the second information represents a change in internal texture of the plurality of objects in the first image and the second image, In step 103, it is determined whether the plurality of objects have been moved based on the first information and the second information.

[0031] Wherein, the first image corresponds to a first image capture time, and the second image corresponds to a second image capture time. It can be understood that determining whether the plurality of objects have moved means determining whether the plurality of objects have moved within a time range from the first image capture time to the second image capture time.

[0032] In actual application, the plurality of objects are stacked, and therefore the plurality of objects are regarded as a single, integrated entity, and when an object located at an edge of the plurality of objects is moved, the outer contour of the plurality of objects in the first and second images changes, and when an object located inside the plurality of objects is moved, the inner texture of the plurality of objects in the first and second images changes, where the inner texture can be understood as information such as the contour and exterior pattern of the object located inside the plurality of objects.

[0033] In step 101, when actually applied, acquiring the first image and the second image of the target area as described above includes: The method may include obtaining a first image and a second image of the target area from an image capture device having a fixed position and image capture angle, the first and second images being captured by the image capture device.

[0034] In actual applications, in some business scenarios, there may be multiple objects placed in multiple positions and stacked, such as cargo piled up in separate piles in a warehouse. To improve image processing efficiency, an image capture device can capture an image including multiple piles of objects, and the electronic device can acquire an image including multiple piles of objects from the image capture device, detect an area where each pile of objects is located from the image, and perform steps 101 to 103 for each of the detected areas, thereby determining whether or not a moved object exists in each pile of objects.

[0035] In light of this, in one embodiment, the aforementioned first image and second image of the target area are obtained by: acquiring a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image corresponding to different image acquisition times; determining, based on the ninth and tenth images, at least one second area in which a plurality of objects placed in a stacked manner are present in the first area; determining a target area from the at least one second area; and cropping the ninth and tenth images based on the target area to obtain the first and second images.

[0036] Here, it can be understood that in the first area there is at least one pile of objects placed in a stacked manner, and in each second area there is one pile of objects placed in a stacked manner.

[0037] In actual application, the electronic device can obtain a ninth image and a tenth image captured by the image capturing device from the image capturing device, where the ninth image corresponds to the first image capturing time and the tenth image corresponds to the second image capturing time. Furthermore, when capturing the ninth and tenth images, the image capturing device may or may not be fixed in position, and the image capturing angle may or may not be fixed, which can be specifically set as needed, and the embodiments of the present application are not limited thereto.

[0038] In actual application, if the position and / or image capture angle of the image capture device is not fixed, it can be understood that the ninth and tenth images need to be processed using a method such as image comparison so that the object in the ninth image corresponds to the object in the tenth image.

[0039] In practical application, a pre-trained model can be used to determine each mountain object in the first region, i.e., to determine at least one second region in the first region.

[0040] In light of this, in one embodiment, determining at least one second region in the first region based on the ninth image and the tenth image as described above includes: The method may include determining at least one second region in the first region using the ninth image, the tenth image, and a third model trained with a target detection algorithm.

[0041] In actual application, the target detection algorithm may include yolo_v5, faster-rcnn, centerNet, etc., and can be specifically set as needed, and the embodiments of the present application are not limited thereto.

[0042] In practical application, the third model needs to be pre-trained. Specifically, a training data set may be determined that may include a predetermined number (e.g., 2000) of images of a predetermined area (which may be set as needed, and multiple objects must be stacked in the area) collected by the image collection device, each mountain object in each image is defined, and coordinate information corresponding to each mountain object is recorded (i.e., labeled). After the labeling is completed, the third model can be trained using the labeled data and the target detection algorithm.

[0043] In actual application, when the position and / or image acquisition angle of the image acquisition device are not fixed, it can be understood that it is necessary to input the ninth image into the third model, obtain at least one candidate second region output by the third model, then input the tenth image into the third model, obtain at least one candidate second region output by the third model, and determine at least one second region by associating the at least one candidate second region output by the third model based on the ninth image with the at least one candidate second region output by the third model based on the tenth image (it is also possible to determine as the second region a second region that is a candidate corresponding to the same mountain object output by the third model both times).

[0044] When the position and image capturing angle of the image capturing device are fixed, only the ninth image needs to be input to the third model to obtain at least one second region output by the third model.

[0045] In practical applications, the second region may be rectangular, and the third model can be used to detect the rectangular region occupied by each mountain object from the image acquired by the image acquisition device, thereby ensuring that information unrelated to external objects does not interfere with the subsequent image processing flow.

[0046] In step 102, if practically applied, the first and second images may be processed using a pre-trained model to determine the first information.

[0047] In light of this, in one embodiment, determining the first information based on the first image and the second image as described above includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image to a first value; and setting pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects (i.e., pixel points corresponding to non-objects) to a second value; and determining the first information by comparing the third image with the fourth image.

[0048] In practical application, the first value may be 1 and the second value may be 0.

[0049] In actual application, the semantic segmentation algorithm may include deeplab_v3, U-net, etc., and can be specifically set as needed, and the embodiments of the present application are not limited thereto.

[0050] In practical application, the first model needs to be pre-trained. Specifically, a training data set may be determined, which may include a predetermined number (e.g., 2000) of images (which may be the same as the images used to train the third model) of a predetermined area (which may be set as needed, and multiple objects stacked in the area) collected by the image collection device, and the coordinate position of the outer contour of each mountain object in each image may be marked. The first model may be trained using the marked data and the semantic segmentation algorithm.

[0051] In one embodiment, determining the first information by comparing the third image with the fourth image includes: determining a plurality of first coefficients based on the third image and the fourth image, each coefficient representing whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; and determining the first information using the plurality of first coefficients.

[0052] In practical application, a specific manner for determining the first coefficient can be set as needed. For example, when a pixel value of a pixel point in the third image is the same as a pixel value in the fourth image, the first coefficient may be equal to 0, and when a pixel value of a pixel point in the third image is different from a pixel value in the fourth image, the first coefficient may be equal to 1.

[0053] In one embodiment, determining the first information using the plurality of first coefficients includes: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; The method may include determining whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have not changed.

[0054] In practical application, the specific method for calculating the second coefficient can be set as needed. It can be understood that the larger the second coefficient, the lower the matching degree between the third image and the fourth image.

[0055] In practical application, the first threshold may be determined by statistically analyzing the effect of the first model on a preset validation data set.

[0056] With this in mind, in one embodiment, the method comprises: determining a first probability, a second probability, a third probability, and a fourth probability of the first model, wherein the first probability represents the probability that the first model will recognize an object in an input image (e.g., the first image and the second image) as an object, i.e., the probability of determining a pixel value of a pixel point corresponding to an object as a first value in the process of binarizing the input image; the second probability represents the probability that the first model will recognize an object in the input image as a non-object, i.e., the probability of determining a pixel value of a pixel point corresponding to an object as a second value in the process of binarizing the input image; the third probability represents the probability that the first model will recognize a non-object in the input image as an object, i.e., the probability of determining a pixel value of a pixel point corresponding to a non-object as a first value in the process of binarizing the input image; and the fourth probability represents the probability that the first model will recognize a non-object in the input image as a non-object, i.e., the probability of determining a pixel value of a pixel point corresponding to a non-object as a second value in the process of binarizing the input image; The method may further include determining the first threshold using the first probability, the second probability, the third probability, and the fourth probability.

[0057] Here, the first probability, the second probability, the third probability, and the fourth probability can be determined by statistically analyzing the effects of the first model in a preset validation data set.

[0058] In practical application, a specific manner for determining the first threshold value using the first probability, the second probability, the third probability, and the fourth probability can be set as needed. For example, when using the first probability, the second probability, the third probability, and the fourth probability, the first threshold value can be determined by combining with ideas such as Bernoulli distribution, binomial distribution, central limit theorem, Gaussian distribution, and 3σ method.

[0059] In step 102, if practically applied, a pre-trained model can be used to process the first and second images to determine the second information.

[0060] In light of this, in one embodiment, determining the second information based on the first image and the second image as described above includes: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image, binarizing the second image using the second model to obtain a sixth image, and setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; and determining the second information using at least the fifth image and the sixth image.

[0061] Here, the first value may be 1 and the second value may be 0.

[0062] In actual application, the edge detection algorithm may include PiDiNet, etc., and can be specifically set as needed, and the embodiments of the present application are not limited thereto.

[0063] In practical applications, the second model may be pre-trained, or may be an open-source model.

[0064] In practical application, the second information may be determined from the third image, the fourth image, the fifth image and the sixth image to further ensure that information unrelated to external objects does not interfere with the subsequent image processing flow.

[0065] In light of this, in one embodiment, determining the second information using at least the fifth image and the sixth image as described above includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values, and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; and determining the second information from the third image, the fourth image, the fifth image, and the sixth image.

[0066] Specifically, in one embodiment, determining the second information from the third image, the fourth image, the fifth image, and the sixth image includes: aligning and multiplying the third image and the fifth image to obtain a seventh image, and aligning and multiplying the fourth image and the sixth image to obtain an eighth image; and determining the second information by comparing the seventh image with the eighth image.

[0067] Here, by aligning and multiplying the third image and the fifth image, and aligning and multiplying the fourth image and the sixth image, interference from information unrelated to external objects can be eliminated. In other words, the seventh image and the eighth image do not contain information unrelated to external objects. By comparing the seventh image and the eighth image to determine the second information, it can ensure that information unrelated to external objects does not interfere with the subsequent image processing flow, and the accuracy of the judgment result can be further improved.

[0068] In actual application, there may be a large number of objects stacked on top of each other. In order to further improve the accuracy of the judgment result, grid division is first performed on the seventh image and the eighth image, and then the grid of the seventh image is compared with the grid of the eighth image on a grid-by-grid basis to determine the local texture change situation of the multiple objects stacked on top of each other.

[0069] In light of this, in one embodiment, determining the second information by comparing the seventh image with the eighth image as described above includes: Dividing the seventh image into a plurality of first grids based on a predetermined rule, and dividing the eighth image into a plurality of second grids based on the predetermined rule; determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a matching degree between one of the first lattices and a corresponding one of the second lattices; and determining the second information using the plurality of third coefficients.

[0070] Here, the predetermined rule can be set as needed. For example, the predetermined rule may include dividing the image into H×W grids, where H and W are integers greater than 0, and H and W may be the same or different, and specific values ​​of H and W may be set based on experience.

[0071] Specifically, in one embodiment, determining the second information by the plurality of third coefficients as described above may include: determining whether each third coefficient is greater than a second threshold; The second information includes: if there is a third coefficient greater than the second threshold, the second information represents that the internal texture in the first image and the second image of the plurality of objects has changed; or if each third coefficient is less than or equal to the second threshold, the second information represents that the internal texture in the first image and the second image of the plurality of objects has not changed.

[0072] In practical application, the second threshold can be determined by statistically analyzing the effect of the second model on a preset validation data set.

[0073] With this in mind, in one embodiment, the method comprises: determining a fifth probability, a sixth probability, a seventh probability, and an eighth probability of the second model, wherein the fifth probability represents the probability that the second model will recognize an edge in an input image (e.g., the first image and the second image) as an edge, i.e., the probability that a pixel value of a pixel point corresponding to an edge will be determined as a first value in the process of binarizing the input image; the sixth probability represents the probability that the second model will recognize an edge in the input image as a non-edge, i.e., the probability that a pixel value of a pixel point corresponding to an edge will be determined as a second value in the process of binarizing the input image; the seventh probability represents the probability that the second model will recognize a non-edge in the input image as an edge, i.e., the probability that a pixel value of a pixel point corresponding to a non-edge will be determined as a first value in the process of binarizing the input image; and the eighth probability represents the probability that the second model will recognize a non-edge in the input image as a non-edge, i.e., the probability that a pixel value of a pixel point corresponding to a non-edge will be determined as a second value in the process of binarizing the input image; The method may further include determining the second threshold using the fifth probability, the sixth probability, the seventh probability, and the eighth probability.

[0074] Here, the fifth, sixth, seventh and eighth probabilities can be determined by statistically analyzing the effects of the second model in a preset validation data set.

[0075] In practical application, a specific manner for determining the second threshold value using the fifth, sixth, seventh, and eighth probabilities can be set as needed. For example, when using the fifth, sixth, seventh, and eighth probabilities, the second threshold value can be determined by combining the fifth, sixth, seventh, and eighth probabilities with ideas such as Bernoulli distribution, binomial distribution, central limit theorem, Gaussian distribution, and 3σ method.

[0076] In one embodiment, in step 103, determining whether the plurality of objects have been moved based on the first information and the second information includes: determining that at least one of the plurality of objects has been moved if the first information indicates that an outer contour of the plurality of objects in the first image and the second image has changed and / or if the second information indicates that an inner texture of the plurality of objects in the first image and the second image has changed; Alternatively, the method may include determining that the plurality of objects have not moved if the first information indicates that the outer contours of the plurality of objects in the first image and the second image have not changed, and the second information indicates that the internal textures of the plurality of objects in the first image and the second image have not changed.

[0077] In practical application, when it is determined that at least one of the plurality of objects has been moved, warning information can be sent to the target device to alert the user that an object has been moved in the target area.

[0078] With this in mind, in one embodiment, the method comprises: The method may further include issuing a warning when it is determined that at least one object among the plurality of objects has been moved; Wherein, the second information is determined by a plurality of third coefficients each representing the degree of matching between one first grid and a corresponding second grid, and when the first image corresponds to a plurality of first grids and the second image corresponds to a plurality of second grids, and the second information represents that the internal textures of the plurality of objects in the first image and the second image have changed, the warning information includes at least one grid marker for locating the moved object, each corresponding to a third coefficient greater than a second threshold.

[0079] In practical applications, the specific recipient of the warning information (i.e., the target device) can be set as needed, and the embodiments of the present application are not limited thereto.

[0080] In practical application, the image processing method according to the embodiment of the present application can monitor whether an object has been moved into a specified area (e.g., a first area). The ninth image may be an image of the original state, i.e., the ninth image may reflect the original state of the object (e.g., the state at the time of entry into storage), and the tenth image may be an image of the current state, i.e., the tenth image may reflect the current state of the object. The ninth image may be updated based on the business status corresponding to the objects (e.g., new cargo has been entered into storage or cargo has been released from storage). The tenth image may be updated periodically or triggered. The periodic updating may include the image capture device capturing a tenth image of the first area at a predetermined period (which may be set as needed, e.g., n seconds, where n is an integer greater than 0) and transmitting the tenth image to the electronic device. The triggered updating may include the electronic device acquiring the tenth image from the image capture device when it receives a detection command from another device (e.g., a terminal).

[0081] An image processing method according to an embodiment of the present application acquires a first image and a second image, the corresponding images being taken at different times, of a target area in which a plurality of objects are stacked; determines, based on the first and second images, first information representing changes in the outer contours of the plurality of objects in the first and second images and second information representing changes in the internal textures of the plurality of objects in the first and second images; and determines, based on the first and second information, whether the plurality of objects have been moved. In an embodiment of the present application, for a target area where multiple objects are stacked, changes in object position are recognized from two perspectives, namely, global and local, based on images taken at different times. In other words, whether multiple objects have been moved is determined based on changes in the outer contours (i.e., global) in the images of the multiple objects and changes in the internal textures (i.e., local) in the images of the multiple objects. In this way, the status of the objects is monitored using computer vision technology (i.e., processing of images of the target area), and whether an object has been moved is intelligently determined, avoiding the waste of human resources caused by manual inspections. Furthermore, compared to manual inspections, by recognizing changes in object position from two perspectives, global and local, subtle object movement events that are difficult for the human eye to notice can be recognized, and the accuracy of the determination results of whether an object has been moved can be improved.

[0082] The present invention will be described in more detail below with reference to application examples.

[0083] This application example provides a computer vision-based surveillance mode for warehouse cargo (i.e., the above-mentioned object), and recognizes changes in cargo from two perspectives: global and local. A global change in cargo refers to a change in the outer contour, while a local change refers to a change in some texture of the corresponding area in the cargo image. Cargo is generally stored in a stacked manner in a warehouse. If cargo at the edge of the cargo area in the image is moved, a change in the contour information at the corresponding position (i.e., the above-mentioned first information) will occur, as shown in the box 201 in FIG. 2. If cargo inside the cargo area in the image is moved, a change in the texture information at the corresponding position (i.e., the above-mentioned second information) will occur, as shown in the box 202 in FIG. 2.

[0084] In this application example, in order to effectively recognize the above phenomenon (i.e., changes in contour information and changes in texture information), the warehouse cargo surveillance mode is as follows: Step 1: detecting cargo regions using a region detection model (i.e., the third model); Step 2: segmenting the cargo area using a segmentation model (i.e., the first model); Step 3: detecting cargo texture using an edge detection model (i.e., the second model); and step 4 of performing final matching.

[0085] Each of the four steps above complements the others, and the specific implementation of each step will be explained below.

[0086] First, a specific implementation of detecting cargo regions using the region detection model (i.e., the third model) will be described.

[0087] In this application example, the role of the region detection model is to detect the rectangular region occupied by the cargo (i.e., the second region) from the images (e.g., the ninth and tenth images) captured by the monitoring camera head (i.e., the image capture device) to ensure that external information unrelated to the cargo does not interfere in the subsequent process. Specifically, as shown in Figure 3, each pile of cargo is defined and a corresponding rectangular frame (e.g., rectangular frame 301) is obtained.

[0088] In this application example, the region detection model uses the yolo_v5 algorithm to detect regions. To train the region detection model, approximately 2,000 pictures captured by the camera head in the warehouse were selected, and each pile of cargo in each picture was defined and the coordinate information corresponding to each rectangular frame was recorded. After labeling was completed, the region detection model was trained using the labeled data. The final region detection model was able to detect each pile of cargo in the newly captured pictures and define the corresponding rectangular frame.

[0089] Next, a specific implementation of dividing the cargo area using the division model (i.e., the first model) will be described.

[0090] In this application example, the role of the segmentation model is to obtain the outer contour of each pile of cargo and determine whether the cargo contour information has changed. The segmentation model processes images using the deeplab_v3 algorithm, and the output of the segmentation model (as shown in Figure 4) is a matrix (e.g., the third and fourth images) whose size is the same as the input picture, where the pixel position corresponding to the cargo in the matrix has a value of 1 (i.e., the first value) and the other parts (i.e., non-cargo parts) have a value of 0 (i.e., the second value).

[0091] In this application example, it is necessary to acquire the marked data in advance and use the marked data to train the segmentation model. The method of acquiring the marked data may include selecting approximately 2,000 pictures captured by the camera head in the warehouse (pictures that have been used to train the region detection model can be used), marking the coordinate position of the outer contour of each pile of cargo in the pictures, and training the segmentation model using the marks. After training is complete, the segmentation model analyzes the newly input picture and generates a 0,1 matrix whose size is the same as the input picture, in which the area corresponding to the cargo is 1 and the rest is 0.

[0092] Thirdly, a concrete implementation of detecting cargo texture using an edge detection model (i.e., the second model) will be described.

[0093] In this application example, the role of the edge detection model is to recognize local textures in the input picture (e.g., the first and second images) and determine whether the texture information has changed from the original. The edge detection model uses PiDiNet to extract important textures by recognizing areas of sudden changes within the picture. The image output by the edge detection model is a 0, 1 matrix, one size of which is the same as the original picture, as shown in Figure 5, with the value of the pixel position corresponding to the edge being 1 (i.e., the first value above) and the values ​​of other positions being 0 (i.e., the second value above).

[0094] In practical applications, the edge detection model can be an open-source model and does not need to be retrained, since edge detection is a general-purpose algorithm and is not limited to detecting cargo edges.

[0095] Fourth, we provide a detailed explanation of the final matching process.

[0096] In this application example, the three steps of cargo area detection, cargo area segmentation, and cargo texture detection all contribute to the final recognition (i.e., final matching) process. After cargo enters the warehouse, an original picture of the cargo is captured by a camera head monitoring the cargo, and then subsequent pictures of the cargo must be taken periodically. Each time a subsequent picture is captured, it must be compared with the original picture to determine whether the cargo has been moved. The original picture of the cargo and the subsequently captured pictures all undergo the three steps of cargo area detection, cargo area segmentation, and cargo texture detection before the final comparison can be performed.

[0097] In practical applications, the position and angle of the surveillance camera head are usually constant in warehouse scenes. Therefore, cargo area detection can be performed only on the original cargo picture, and the obtained detection frame can also be used for the subsequently acquired cargo picture. However, both the original cargo picture and the subsequently acquired cargo picture need to undergo cargo area segmentation and cargo texture detection.

[0098] In practical application, when the cargo status is updated, for example, when new cargo is received or released from the warehouse, the original picture can be updated.

[0099] In this application example, after completing the above steps (i.e., cargo area detection, cargo area segmentation, and cargo texture detection), the original cargo picture (

number

number

number

number

number

number

number

[0100] 1) Subpicture

number

number

number

number

[0101] 2) Calculate the coefficient of variation (i.e., the second coefficient) using the following formula:

number

number

number

number

number

number

number

number

[0102] 3) If the coefficient of variation is greater than a certain threshold (ie, the first threshold), it is determined that one load of cargo corresponding to the sub-picture has moved.

[0103] The comparison process for texture information includes the following steps.

[0104] 1) Subpicture

number

number

number

number

[0105] 2) Subpicture

number

number

number

number

[0106] 3)

number

number

number

number

number

number

[0107] 4)

number

number

number

number

[0108] 5) Calculate the coefficient of variation (i.e., the third coefficient) for each mesh using the following formula:

number

number

[0109] The reason for dividing the area into meshes and then calculating the coefficient of variation is that movement may occur only in a certain area, and if the entire picture is directly compared, the difference obtained is small, making it difficult to accurately determine whether cargo has moved or not. Dividing the area into meshes effectively solves this problem. Dividing the area into meshes also helps to locate the specific location where movement has occurred.

[0110] 6) If the coefficient of variation is greater than a certain threshold (ie, the second threshold), it is determined that the corresponding part (ie, region) in the corresponding mesh has moved.

[0111] As can be seen from the above steps, both contour and texture comparison require comparing the coefficient of variation with a corresponding threshold. However, setting the threshold is usually a difficult problem. Therefore, in this application example, the threshold is derived by statistical means to help make the judgment more accurate.

[0112] In this application example, when comparing the contour information, the threshold value (ie, the first threshold value) is derived as follows: The probability that the segmentation model recognizes the cargo part as the cargo part, the probability that the cargo part is recognized as the other part, the probability that the other part is recognized as the cargo part, and the probability that the other part is recognized as the other part are obtained as follows:

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0113] In this application example, when comparing texture information, the threshold value (ie, the second threshold value) is derived as follows: The probability that the edge detection model recognizes an edge as an edge, the probability that it recognizes an edge as a non-edge, the probability that it recognizes a non-edge as an edge, and the probability that it recognizes a non-edge as a non-edge are obtained, respectively.

number

number

number

number

number

number

number

number

number

number

number

number

number

[0114] In this application example, the specific flow of the image processing process is as follows.

[0115] 1) Model training: Data is labeled according to a preset rule, and a region detection model and a segmentation model are trained to obtain an edge detection model. At the same time, the probability values ​​corresponding to the segmentation model and the edge detection model are calculated from the validation data set.

number

number

[0116] 2) Detecting cargo area: After completing the storage or status update of cargo, the corresponding camera head will capture the original picture of the cargo, and the detection model will be used to obtain the detection frame corresponding to each pile of cargo.

[0117] 3) Cargo area segmentation: Cargo area segmentation is performed on the original cargo picture to obtain the corresponding segmentation diagram. If the cargo status is not updated, this operation only needs to be performed once. Cargo area segmentation is performed on periodically captured cargo pictures to obtain the corresponding segmentation diagram. This needs to be performed once every time a new picture is captured.

[0118] 4) Cargo texture detection: An edge detection algorithm is used to detect texture on the original cargo picture to obtain the corresponding edge picture. This operation only needs to be performed once when the cargo state is not updated; edge detection is performed on periodically captured cargo pictures to obtain the corresponding edge picture. This needs to be performed once every time a new picture is captured.

[0119] 5) Contrast the contours. Sub-pictures of each pair

number

number

number

number

[0120] 6) Contrast textures. Subpictures of each pair.

number

number

number

[0121] An aspect of this application example is to use computer vision technology to monitor cargo in the warehouse using surveillance camera heads in the warehouse and determine whether it has been moved, and to use image segmentation and edge detection technology to obtain the contour and texture information of the cargo, and based on these two types of information, compare the state (i.e., position) of the cargo to determine whether it has been moved from two perspectives: global and local, and further use statistical methods to derive and obtain a threshold value for the difference between the original state of the cargo and the state of the cargo after it has been collected, which helps to more accurately determine whether the cargo has been moved.

[0122] This application example uses computer vision technology to monitor the status (i.e., location) of warehouse cargo. If a change in the cargo status (i.e., location) is detected, an alarm is issued promptly and a request for human inspection is made. Because warehouses typically have a large number of surveillance camera heads, this method can fully utilize existing resources and effectively reduce manpower consumption. At the same time, computer vision can also recognize subtle changes that are difficult for the human eye to notice.

[0123] To realize the method of the embodiment of the present application, the embodiment of the present application further provides an image processing device provided in an electronic device (for example, attached to a server), and as shown in FIG. 6, the device includes: a first processing unit 601 configured to acquire a first image and a second image of a target area where a plurality of objects are present in a stacked manner, the first image and the second image having different corresponding image acquisition times; a second processing unit 602 configured to determine, based on the first and second images, first information representing changes in outer contours of the plurality of objects in the first and second images and second information representing changes in inner textures of the plurality of objects in the first and second images; and a third processing unit 603 configured to determine whether the plurality of objects have been moved based on the first information and the second information.

[0124] Wherein, in one embodiment, the second processing unit 602 further comprises: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values ​​and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; The first information is determined by comparing the third image with the fourth image.

[0125] In one embodiment, the second processing unit 602 further comprises: determining a plurality of first coefficients based on the third image and the fourth image, each coefficient representing whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; The plurality of first coefficients are configured to determine the first information.

[0126] In one embodiment, the second processing unit 602 further comprises: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; The method is configured to determine whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have not changed.

[0127] In one embodiment, the second processing unit 602 further comprises: configured to determine a first probability, a second probability, a third probability, and a fourth probability of the first model, and to determine the first threshold using the first probability, the second probability, the third probability, and the fourth probability; The first probability represents the probability that the first model recognizes an object in the input image as an object, the second probability represents the probability that the first model recognizes an object in the input image as a non-object, the third probability represents the probability that the first model recognizes a non-object in the input image as an object, and the fourth probability represents the probability that the first model recognizes a non-object in the input image as a non-object.

[0128] In one embodiment, the second processing unit 602 further comprises: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image; binarizing the second image using the second model to obtain a sixth image; setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; The second information is determined by at least the fifth image and the sixth image.

[0129] In one embodiment, the second processing unit 602 further comprises: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values ​​and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; The third image, the fourth image, the fifth image, and the sixth image are configured to determine the second information.

[0130] In one embodiment, the second processing unit 602 further comprises: aligning and multiplying the third image and the fifth image to obtain a seventh image; aligning and multiplying the fourth image and the sixth image to obtain an eighth image; The seventh image and the eighth image are compared to determine the second information.

[0131] In one embodiment, the second processing unit 602 further comprises: Dividing the seventh image into a plurality of first grids based on a predetermined rule, dividing the eighth image into a plurality of second grids based on the predetermined rule, determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a matching degree between one of the first lattices and a corresponding one of the second lattices; The plurality of third coefficients are configured to determine the second information.

[0132] In one embodiment, the second processing unit 602 further comprises: determining whether each third coefficient is greater than a second threshold; The second information is configured to indicate that if there is a third coefficient greater than the second threshold, the internal texture in the first image and the second image of the plurality of objects has changed, or if each third coefficient is less than or equal to the second threshold, the second information indicates that the internal texture in the first image and the second image of the plurality of objects has not changed.

[0133] In one embodiment, the second processing unit 602 further comprises: determining a fifth probability, a sixth probability, a seventh probability, and an eighth probability of the second model; configured to determine the second threshold using the fifth probability, the sixth probability, the seventh probability, and the eighth probability; The fifth probability represents the probability that the second model will recognize an edge in the input image as an edge, the sixth probability represents the probability that the second model will recognize an edge in the input image as a non-edge, the seventh probability represents the probability that the second model will recognize a non-edge in the input image as an edge, and the eighth probability represents the probability that the second model will recognize a non-edge in the input image as a non-edge.

[0134] In one embodiment, the third processing unit 603 further comprises: determining that at least one of the plurality of objects has been moved when the first information indicates that an outer contour of the plurality of objects in the first image and the second image has changed, and / or when the second information indicates that an inner texture of the plurality of objects in the first image and the second image has changed; Alternatively, the method is configured to determine that the plurality of objects have not moved if the first information indicates that the outer contours of the plurality of objects in the first image and the second image have not changed, and the second information indicates that the internal textures of the plurality of objects in the first image and the second image have not changed.

[0135] In one embodiment, the device further comprises a communication unit, and the third processing unit 603 is further configured to issue warning information through the communication module when determining that at least one object among the plurality of objects has been moved, wherein: The second information is determined by a plurality of third coefficients, each of which represents a degree of matching between one first grid and a corresponding second grid, and when the first image corresponds to a plurality of first grids and the second image corresponds to a plurality of second grids and the second information represents that the internal texture of the plurality of objects in the first image and the second image has changed, the warning information includes at least one grid marker for locating the moved object, each of which corresponds to a third coefficient greater than a second threshold.

[0136] In one embodiment, the first processing unit 601 further comprises: Acquire a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image being acquired at different times; determining, based on the ninth and tenth images, at least one second region in which a plurality of objects placed in a stacked manner are present in the first region; The method is configured to determine a target area from the at least one second area, and crop the ninth and tenth images based on the target area to obtain the first and second images.

[0137] In one embodiment, the first processing unit 601 is further configured to determine at least one second region in the first region using the ninth image, the tenth image and a third model trained by a target detection algorithm.

[0138] In practical application, the communication unit can be realized by a communication interface in an image processing device, and the first processing unit 601, the second processing unit 602 and the third processing unit 603 can be realized by a processor in the image processing device.

[0139] Although the image processing device according to the above embodiment has been described using only the partitions of the above program modules as an example when processing an image, in actual application, the above processes may be assigned to be completed by different program modules as needed, i.e., the internal structure of the device may be partitioned into different program modules to complete all or part of the processes described above. Furthermore, the image processing device according to the above embodiment belongs to the same concept as the image processing method embodiment, and the details of the specific implementation process thereof are to be referred to the method embodiment, and will not be repeated here.

[0140] In order to implement the above program modules based on hardware and to implement the methods of the embodiments of the present application, the embodiments of the present application further provide an electronic device, as shown in FIG. 7, the electronic device 700 includes: a communication interface 701 capable of performing information interaction with other electronic devices; a processor 702 connected to the communication interface 701 to realize information interaction with other electronic devices, and used to execute the method according to one or more of the above technical aspects when a computer program is running; and a memory 703 for storing a computer program operable by the processor 702.

[0141] Specifically, the processor 702: Acquire a first image and a second image of a target area where a plurality of objects placed in a stacked manner exist, the first image and the second image being acquired at different corresponding image acquisition times; determining first information representing a change in an outer contour of the plurality of objects in the first image and the second image and second information representing a change in an inner texture of the plurality of objects in the first image and the second image based on the first image and the second image; The system is configured to determine whether the plurality of objects have been moved based on the first information and the second information.

[0142] In one embodiment, the processor 702 further comprises: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values ​​and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; The first information is determined by comparing the third image with the fourth image.

[0143] In one embodiment, the processor 702 further comprises: determining a plurality of first coefficients based on the third image and the fourth image, each coefficient representing whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; The plurality of first coefficients are configured to determine the first information.

[0144] In one embodiment, the processor 702 further comprises: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; The method is configured to determine whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have not changed.

[0145] In one embodiment, the processor 702 further comprises: configured to determine a first probability, a second probability, a third probability, and a fourth probability of the first model, and to determine the first threshold using the first probability, the second probability, the third probability, and the fourth probability; The first probability represents the probability that the first model recognizes an object in the input image as an object, the second probability represents the probability that the first model recognizes an object in the input image as a non-object, the third probability represents the probability that the first model recognizes a non-object in the input image as an object, and the fourth probability represents the probability that the first model recognizes a non-object in the input image as a non-object.

[0146] In one embodiment, the processor 702 further comprises: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image; binarizing the second image using the second model to obtain a sixth image; setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; The second information is determined by at least the fifth image and the sixth image.

[0147] In one embodiment, the processor 702 further comprises: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values ​​and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; The third image, the fourth image, the fifth image, and the sixth image are configured to determine the second information.

[0148] In one embodiment, the processor 702 further comprises: aligning and multiplying the third image and the fifth image to obtain a seventh image; aligning and multiplying the fourth image and the sixth image to obtain an eighth image; The seventh image and the eighth image are compared to determine the second information.

[0149] In one embodiment, the processor 702 further comprises: Dividing the seventh image into a plurality of first grids based on a predetermined rule, dividing the eighth image into a plurality of second grids based on the predetermined rule, determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a matching degree between one of the first lattices and a corresponding one of the second lattices; The plurality of third coefficients are configured to determine the second information.

[0150] In one embodiment, the processor 702 further comprises: determining whether each third coefficient is greater than a second threshold; The second information is configured to indicate that if there is a third coefficient greater than the second threshold, the internal texture in the first image and the second image of the plurality of objects has changed, or if each third coefficient is less than or equal to the second threshold, the second information indicates that the internal texture in the first image and the second image of the plurality of objects has not changed.

[0151] In one embodiment, the processor 702 further comprises: determining a fifth probability, a sixth probability, a seventh probability, and an eighth probability of the second model; configured to determine the second threshold using the fifth probability, the sixth probability, the seventh probability, and the eighth probability; The fifth probability represents the probability that the second model will recognize an edge in the input image as an edge, the sixth probability represents the probability that the second model will recognize an edge in the input image as a non-edge, the seventh probability represents the probability that the second model will recognize a non-edge in the input image as an edge, and the eighth probability represents the probability that the second model will recognize a non-edge in the input image as a non-edge.

[0152] In one embodiment, the processor 702 further comprises: determining that at least one of the plurality of objects has been moved when the first information indicates that an outer contour of the plurality of objects in the first image and the second image has changed, and / or when the second information indicates that an inner texture of the plurality of objects in the first image and the second image has changed; Alternatively, the method is configured to determine that the plurality of objects have not moved if the first information indicates that the outer contours of the plurality of objects in the first image and the second image have not changed, and the second information indicates that the internal textures of the plurality of objects in the first image and the second image have not changed.

[0153] In one embodiment, the processor 702 is further configured to issue warning information via the communication interface 701 when determining that at least one object among the plurality of objects has been moved, wherein: The second information is determined by a plurality of third coefficients, each of which represents a degree of matching between one first grid and a corresponding second grid, and when the first image corresponds to a plurality of first grids and the second image corresponds to a plurality of second grids and the second information represents that the internal texture of the plurality of objects in the first image and the second image has changed, the warning information includes at least one grid marker for locating the moved object, each of which corresponds to a third coefficient greater than a second threshold.

[0154] In one embodiment, the processor 702 further comprises: Acquire a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image being acquired at different times; determining, based on the ninth and tenth images, at least one second region in which a plurality of objects placed in a stacked manner are present in the first region; The method is configured to determine a target area from the at least one second area, and crop the ninth and tenth images based on the target area to obtain the first and second images.

[0155] In one embodiment, the processor 702 is further configured to determine at least one second region in the first region using the ninth image, the tenth image, and a third model trained with a target detection algorithm.

[0156] For details of the specific process of the processor 702 performing the above operations, please refer to the method embodiments and will not be described again here.

[0157] Of course, in actual applications, the components in the electronic device 700 are coupled together via a bus system 704. It will be understood that the bus system 704 is used to realize communication connections between these components. The bus system 704 includes not only a data bus but also a power bus, a control bus, and a status signal bus. However, for clarity, various buses are labeled as the bus system 704 in FIG. 7.

[0158] Memory 703 in this embodiment is used to store various types of data to support the operation of electronic device 700. Examples of this data include any computer programs intended to be operated on electronic device 700.

[0159] The methods disclosed in the above-described embodiments of the present application may be applied to or implemented by the processor 702. The processor 702 may be an integrated circuit chip having signal processing capabilities. In the implementation process, each step of the above-described method may be completed by an integrated logic circuit in hardware in the processor 702 or by instructions in the form of software. The processor 702 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor 702 may implement or execute each method, step, and logic block diagram disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application may be directly implemented by a hardware decoding processor or a combination of hardware and software modules of the decoding processor. The software modules may be located in a storage medium. The storage medium is located in memory 703, and processor 702 reads the information in memory 703 and completes the steps of the aforementioned method in combination with its hardware.

[0160] In an exemplary embodiment, electronic device 700 may be implemented with one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), general-purpose processors, controllers, Micro Controller Units (MCUs), Microprocessors, or other electronic components to perform the methods.

[0161] It is understood that memory 703 in the present embodiment may be volatile or nonvolatile memory, or may include both volatile and nonvolatile memory. Nonvolatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM). Magnetic surface memory may be magnetic disk memory or magnetic tape memory. Volatile memory may be random access memory (RAM) used as an external cache.By way of example and not limitation, many types of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), and direct Rambus random access memory (DRRAM). The memory described in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0162] In an exemplary embodiment, the present application further provides a storage medium, i.e., a computer storage medium, which is specifically a computer-readable storage medium, such as a memory 703 that stores a computer program, which is executable by the processor 702 of the electronic device 700 to complete the steps of the above-described method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM.

[0163] It should be noted that terms such as "first," "second," etc. need not be used to describe a particular order or sequence, but are merely used to distinguish between similar objects.

[0164] Furthermore, the technical aspects described in the examples of the present application can be combined in any manner as long as they do not conflict with each other.

[0165] The above are only preferred embodiments of the present application, and do not limit the scope of protection of the present application. The inventions disclosed herein include the following: [Aspect 1] acquiring a first image and a second image of a target area where a plurality of objects are stacked, the first image and the second image being captured at different times; determining, based on the first and second images, first information representing a change in the outer contour of the plurality of objects in the first and second images and second information representing a change in the inner texture of the plurality of objects in the first and second images; determining whether the plurality of objects have been moved based on the first information and the second information. Image processing methods. [Aspect 2] The step of determining the first information based on the first image and the second image includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values, and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; and determining the first information by comparing the third image with the fourth image. 2. The image processing method according to embodiment 1. [Aspect 3] The step of determining the first information by comparing the third image with the fourth image includes: determining a plurality of first coefficients based on the third image and the fourth image, each coefficient representing whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; determining the first information using the plurality of first coefficients; 3. An image processing method according to aspect 2. [Aspect 4] The step of determining the first information using the plurality of first coefficients includes: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; determining whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have not changed. An image processing method according to aspect 3. [Aspect 5] determining a first probability, a second probability, a third probability, and a fourth probability of the first model, wherein the first probability represents a probability that the first model recognizes an object in the input image as an object, the second probability represents a probability that the first model recognizes an object in the input image as a non-object, the third probability represents a probability that the first model recognizes a non-object in the input image as an object, and the fourth probability represents a probability that the first model recognizes a non-object in the input image as a non-object; determining the first threshold value using the first probability, the second probability, the third probability, and the fourth probability. An image processing method according to embodiment 4. [Aspect 6] The step of determining the second information based on the first image and the second image includes: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image, binarizing the second image using the second model to obtain a sixth image, and setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; determining the second information from at least the fifth image and the sixth image; 2. The image processing method according to embodiment 1. [Aspect 7] The step of determining the second information using at least the fifth image and the sixth image includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values, and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; determining the second information from the third image, the fourth image, the fifth image, and the sixth image; An image processing method according to embodiment 6. [Aspect 8] The above-mentioned determining the second information from the third image, the fourth image, the fifth image, and the sixth image includes: aligning and multiplying the third image and the fifth image to obtain a seventh image, and aligning and multiplying the fourth image and the sixth image to obtain an eighth image; and determining the second information by comparing the seventh image with the eighth image. An image processing method according to embodiment 7. [Aspect 9] The step of determining the second information by comparing the seventh image with the eighth image includes: Dividing the seventh image into a plurality of first grids based on a predetermined rule, and dividing the eighth image into a plurality of second grids based on the predetermined rule; determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a matching degree between one of the first lattices and a corresponding one of the second lattices; determining the second information by the plurality of third coefficients; An image processing method according to embodiment 8. [Aspect 10] The step of determining the second information using the plurality of third coefficients includes: determining whether each third coefficient is greater than a second threshold; When there is a third coefficient greater than the second threshold, the second information represents that an internal texture of the plurality of objects in the first image and the second image has changed, or when each third coefficient is less than or equal to the second threshold, the second information represents that an internal texture of the plurality of objects in the first image and the second image has not changed. An image processing method according to embodiment 9. [Aspect 11] determining a fifth probability, a sixth probability, a seventh probability, and an eighth probability of the second model, wherein the fifth probability represents the probability that the second model recognizes an edge in the input image as an edge, the sixth probability represents the probability that the second model recognizes an edge in the input image as a non-edge, the seventh probability represents the probability that the second model recognizes a non-edge in the input image as an edge, and the eighth probability represents the probability that the second model recognizes a non-edge in the input image as a non-edge; determining the second threshold using the fifth probability, the sixth probability, the seventh probability, and the eighth probability. An image processing method according to embodiment 10. [Aspect 12] The step of determining whether the plurality of objects have been moved based on the first information and the second information includes: determining that at least one of the plurality of objects has been moved if the first information indicates that an outer contour of the plurality of objects in the first image and the second image has changed and / or if the second information indicates that an inner texture of the plurality of objects in the first image and the second image has changed; or determining that the plurality of objects have not moved when the first information indicates that the outer contours of the plurality of objects in the first image and the second image have not changed and the second information indicates that the inner textures of the plurality of objects in the first image and the second image have not changed. 12. The image processing method according to any one of aspects 1 to 11. [Aspect 13] and issuing a warning when it is determined that at least one object among the plurality of objects has been moved; the second information is determined by a plurality of third coefficients each representing a matching degree between one first grid and a corresponding second grid, the first image corresponds to a plurality of first grids, the second image corresponds to a plurality of second grids, and when the second information represents that internal textures of the plurality of objects in the first image and the second image have changed, the warning information includes at least one grid mark for locating the moved object, each of the third coefficients corresponding to a second threshold value greater than one; 13. An image processing method according to embodiment 12. [Aspect 14] The aforementioned acquiring the first image and the second image of the target area includes: acquiring a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image corresponding to different image acquisition times; determining, based on the ninth and tenth images, at least one second area in which a plurality of objects placed in a stacked manner are present in the first area; determining a target area from the at least one second area; and cropping the ninth and tenth images based on the target area to obtain the first and second images. 12. The image processing method according to any one of aspects 1 to 11. [Aspect 15] The step of determining at least one second region in the first region based on the ninth image and the tenth image includes: determining at least one second region in the first region using the ninth image, the tenth image, and a third model trained with a target detection algorithm; An image processing method according to embodiment 14. [Aspect 16] a first processing unit configured to acquire a first image and a second image of a target area where a plurality of objects are present in a stacked manner, the first image and the second image having different corresponding image acquisition times; a second processing unit configured to determine, based on the first and second images, first information representing changes in outer contours of the plurality of objects in the first and second images and second information representing changes in inner textures of the plurality of objects in the first and second images; a third processing unit configured to determine whether the plurality of objects have been moved based on the first information and the second information. Image processing device. [Aspect 17] a processor and a memory used to store a computer program executable by the processor; When the processor runs the computer program, it is used to perform the steps of the method according to any one of aspects 1 to 15. Electronic machines. [Aspect 18] A computer program is stored that, when executed by a processor, implements the steps of the method according to any one of aspects 1 to 15. storage medium.

Claims

1. acquiring a first image and a second image of a target area where a plurality of objects are stacked, the first image and the second image being captured at different times; determining, based on the first and second images, first information representing a change in outer contours of the plurality of objects in the first and second images and second information representing a change in internal textures of the plurality of objects in the first and second images; determining whether the plurality of objects have been moved based on the first information and the second information; The step of determining the second information based on the first image and the second image includes: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image; binarizing the second image using the second model to obtain a sixth image; and setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; determining the second information from at least the fifth image and the sixth image; Image processing methods.

2. The step of determining the first information based on the first image and the second image includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image to a first value; and setting pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects to a second value; and determining the first information by comparing the third image with the fourth image. The image processing method according to claim 1 .

3. The step of determining the first information by comparing the third image with the fourth image includes: determining a plurality of first coefficients based on the third image and the fourth image, each coefficient representing whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; determining the first information using the plurality of first coefficients; The image processing method according to claim 2 .

4. The step of determining the first information using the plurality of first coefficients includes: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; determining whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that the outer contours of the plurality of objects in the first image and the second image have not changed. The image processing method according to claim 3 .

5. determining a first probability, a second probability, a third probability, and a fourth probability of the first model, wherein the first probability represents a probability that the first model recognizes an object in the input image as an object, the second probability represents a probability that the first model recognizes an object in the input image as a non-object, the third probability represents a probability that the first model recognizes a non-object in the input image as an object, and the fourth probability represents a probability that the first model recognizes a non-object in the input image as a non-object; determining the first threshold value using the first probability, the second probability, the third probability, and the fourth probability. The image processing method according to claim 4.

6. The step of determining the second information using at least the fifth image and the sixth image includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image to a first value; and setting pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects to a second value; determining the second information from the third image, the fourth image, the fifth image, and the sixth image; The image processing method according to claim 1 .

7. The step of determining the second information using the third image, the fourth image, the fifth image, and the sixth image includes the steps of: multiplying the third image and the fifth image in registration to obtain a seventh image, and multiplying the fourth image and the sixth image in registration to obtain an eighth image; and determining the second information by comparing the seventh image with the eighth image. The image processing method according to claim 6.

8. The step of determining the second information by comparing the seventh image with the eighth image includes: Dividing the seventh image into a plurality of first grids based on a predetermined rule, and dividing the eighth image into a plurality of second grids based on the predetermined rule; determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a matching degree between one of the first lattices and a corresponding one of the second lattices; determining the second information using the plurality of third coefficients; The image processing method according to claim 7.

9. The step of determining the second information using the plurality of third coefficients includes: determining whether each third coefficient is greater than a second threshold; When a third coefficient exists that is greater than the second threshold, the second information represents that an internal texture of the plurality of objects in the first image and the second image has changed; or when each third coefficient is smaller than or equal to the second threshold, the second information represents that an internal texture of the plurality of objects in the first image and the second image has not changed. The image processing method according to claim 8.

10. determining a fifth probability, a sixth probability, a seventh probability, and an eighth probability of the second model, wherein the fifth probability represents the probability that the second model recognizes an edge in the input image as an edge, the sixth probability represents the probability that the second model recognizes an edge in the input image as a non-edge, the seventh probability represents the probability that the second model recognizes a non-edge in the input image as an edge, and the eighth probability represents the probability that the second model recognizes a non-edge in the input image as a non-edge; determining the second threshold value using the fifth, sixth, seventh, and eighth probabilities. The image processing method according to claim 9.

11. The step of determining whether the plurality of objects have been moved based on the first information and the second information includes: determining that at least one of the plurality of objects has been moved if the first information indicates that an outer contour of the plurality of objects in the first image and the second image has changed and / or if the second information indicates that an inner texture of the plurality of objects in the first image and the second image has changed; or determining that the plurality of objects has not moved when the first information indicates that outer contours of the plurality of objects in the first image and the second image have not changed, and the second information indicates that inner textures of the plurality of objects in the first image and the second image have not changed. The image processing method according to claim 1 .

12. and issuing a warning when it is determined that at least one object among the plurality of objects has been moved; the second information is determined by a plurality of third coefficients each representing a degree of matching between one first grid and a corresponding second grid, the first image corresponds to a plurality of first grids, the second image corresponds to a plurality of second grids, and when the second information represents that internal textures of the plurality of objects in the first image and the second image have changed, the warning information includes at least one grid mark for locating the moved object, each of which corresponds to a third coefficient greater than a second threshold value; The image processing method according to claim 11.

13. The step of acquiring the first and second images of the target area includes: acquiring a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image corresponding to different image acquisition times; determining, based on the ninth and tenth images, at least one second area in which a plurality of objects placed in a stacked manner are present in the first area; determining a target area from the at least one second area; and cropping the ninth and tenth images based on the target area to obtain the first and second images. The image processing method according to claim 1 .

14. The step of determining at least one second region in the first region based on the ninth image and the tenth image includes: determining at least one second region in the first region using the ninth image, the tenth image, and a third model trained with a target detection algorithm; The image processing method according to claim 13.

15. a first processing unit configured to acquire a first image and a second image of a target area where a plurality of objects are present in a stacked manner, the first image and the second image having different corresponding image acquisition times; a second processing unit configured to determine, based on the first and second images, first information representing changes in outer contours of the plurality of objects in the first and second images and second information representing changes in inner textures of the plurality of objects in the first and second images; a third processing unit configured to determine whether the plurality of objects have been moved based on the first information and the second information; The second processing unit further comprises: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image; binarizing the second image using the second model to obtain a sixth image; setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; configured to determine the second information by at least the fifth image and the sixth image; Image processing device.

16. a processor and a memory used to store a computer program executable by the processor; The processor, when running the computer program, is used to perform the steps of the method according to any one of claims 1 to 14. electronic equipment.

17. A computer program is stored which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14. storage medium.

Images