IMAGE PROCESSING METHOD, APPARATUS, ELECTRONIC DEVICE, AND STORAGE MEDIUM

Abstract

The present application discloses an image processing method, device, electronic device and storage medium, which are related to the technical field of artificial intelligence, and includes the steps of: acquiring a first image and a second image of a target area where a plurality of objects are placed in a stacked manner, the first image and the second image are captured at different times; determining, based on the first image and the second image, first information representing changes in the outer contours of the plurality of objects in the first image and the second image and second information representing changes in the inner textures of the plurality of objects in the first image and the second image; and determining, based on the first information and the second information, whether the plurality of objects have been moved.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is filed based on and claims priority to a Chinese patent application having application number 202210350472.4 and filing date April 2, 2022, the entire contents of which are hereby incorporated by reference into this application.

[0002] The present application relates to the technical field of artificial intelligence, and in particular to an image processing method, device, electronic device, and storage medium. [Background technology]

[0003] In some business scenarios, it is necessary to determine whether an object has been moved. For example, currently, supply chain finance is an important business innovation direction in the current logistics field. In the supply chain finance business, logistics companies and banks cooperate with each other to gain mutual trust and provide services to businesses in the supply chain field. One of the important services is that businesses apply for loans by putting their goods on the bank as collateral, and since logistics companies have advantages in terms of warehousing, the logistics companies provide a place for the goods to be collateralized. In this process, one of the important tasks of logistics companies is to ensure the safety of the goods, that is, to ensure that the goods are not moved out of the warehouse by anyone or in any way when not authorized. Only in this way can we provide reliable services to businesses and banks and ensure the smooth operation of supply chain finance business.

[0004] However, in the related art, there is still no effective solution for how to intelligently determine whether an object has been moved. Summary of the Invention [Problem to be solved by the invention]

[0005] To solve the problems in the related art, embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium. [Means for solving the problem]

[0006] The technical aspects of the embodiments of the present application are realized as follows.

[0007] The embodiment of the present application is acquiring a first image and a second image of a target area in which a plurality of objects are placed in a stacked manner, the first image and the second image having different corresponding image acquisition times; determining, based on the first and second images, first information representing a change in an outer contour of the plurality of objects in the first and second images and second information representing a change in an inner texture of the plurality of objects in the first and second images; determining whether the plurality of objects have been moved based on the first information and the second information.

[0008] In the above aspect, determining the first information based on the first image and the second image includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image, binarizing the second image using the first model to obtain a fourth image, setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as a first value, and setting pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as a second value; and determining the first information by comparing the third image with the fourth image.

[0009] In the above aspect, determining the first information by comparing the third image with the fourth image includes: determining a plurality of first coefficients based on the third image and the fourth image, each of the first coefficients representing whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; determining the first information using the plurality of first coefficients.

[0010] In the above aspect, determining the first information by the plurality of first coefficients includes: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; determining whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that the outer contours of the multiple objects in the first and second images have changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that the outer contours of the multiple objects in the first and second images have not changed.

[0011] In the above aspect, the method further comprises: determining a first probability, a second probability, a third probability and a fourth probability of the first model, the first probability representing the probability that the first model will recognize an object in the input image as an object, the second probability representing the probability that the first model will recognize an object in the input image as a non-object, the third probability representing the probability that the first model will recognize a non-object in the input image as an object, and the fourth probability representing the probability that the first model will recognize a non-object in the input image as a non-object; The method further includes determining the first threshold using the first probability, the second probability, the third probability, and the fourth probability.

[0012] In the above aspect, determining the second information based on the first image and the second image includes: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image, binarizing the second image using the second model to obtain a sixth image, and setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; and determining the second information from at least the fifth image and the sixth image.

[0013] In the above aspect, determining the second information by at least the fifth image and the sixth image includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image, binarizing the second image using the first model to obtain a fourth image, setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as a first value, and setting pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as a second value; determining the second information using the third image, the fourth image, the fifth image and the sixth image.

[0014] In the above aspect, determining the second information by the third image, the fourth image, the fifth image, and the sixth image includes: aligning and multiplying the third image and the fifth image to obtain a seventh image, and aligning and multiplying the fourth image and the sixth image to obtain an eighth image; and determining the second information by comparing the seventh image with the eighth image.

[0015] In the above aspect, determining the second information by comparing the seventh image with the eighth image includes: Dividing the seventh image according to a preset rule to obtain a plurality of first grids, and dividing the eighth image according to the preset rule to obtain a plurality of second grids; determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a matching degree between a first lattice and a corresponding second lattice; determining the second information using the plurality of third coefficients.

[0016] In the above aspect, determining the second information by the plurality of third coefficients includes: determining whether each third coefficient is greater than a second threshold; When a third coefficient exists that is greater than the second threshold, the second information indicates that the internal texture in the first image and the second image of the plurality of objects has changed, or when each third coefficient is less than or equal to the second threshold, the second information indicates that the internal texture in the first image and the second image of the plurality of objects has not changed.

[0017] In the above aspect, the method further comprises: determining a fifth probability, a sixth probability, a seventh probability and an eighth probability of the second model, the fifth probability representing the probability that the second model will recognize an edge in the input image as an edge, the sixth probability representing the probability that the second model will recognize an edge in the input image as a non-edge, the seventh probability representing the probability that the second model will recognize a non-edge in the input image as an edge, and the eighth probability representing the probability that the second model will recognize a non-edge in the input image as a non-edge; The method further includes determining the second threshold using the fifth probability, the sixth probability, the seventh probability, and the eighth probability.

[0018] In the above aspect, determining whether the plurality of objects have been moved based on the first information and the second information includes: determining that at least one of the plurality of objects has been moved if the first information indicates that an outer contour of the plurality of objects in the first and second images has changed and / or if the second information indicates that an inner texture of the plurality of objects in the first and second images has changed; Or, determining that the plurality of objects has not moved when the first information indicates that the outer contours of the plurality of objects in the first image and the second image have not changed and the second information indicates that the internal textures of the plurality of objects in the first image and the second image have not changed.

[0019] In the above aspect, the method further comprises: and issuing a warning when it is determined that at least one object of the plurality of objects has been moved; Wherein, the second information is determined by a plurality of third coefficients each representing a degree of matching between a first grid and a corresponding second grid, and when the first image corresponds to a plurality of first grids and the second image corresponds to a plurality of second grids, and the second information represents that internal textures in the first and second images of the plurality of objects have changed, the warning information includes at least one grid marker for locating a moved object, each corresponding to a third coefficient greater than a second threshold value.

[0020] In the above aspect, acquiring the first image and the second image of the target area includes: acquiring a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image having different corresponding image acquisition times; determining, based on the ninth and tenth images, at least one second region in which a plurality of objects are present in a stacked manner in the first region; determining a target area from the at least one second area; and cropping the ninth and tenth images based on the target area to obtain the first and second images.

[0021] In the above aspect, determining at least one second region in the first region based on the ninth image and the tenth image includes: determining at least one second region in the first region using the ninth image, the tenth image and a third model trained with a target detection algorithm.

[0022] The embodiment of the present application is a first processing unit configured to acquire a first image and a second image of a target area in which a plurality of objects are present, the first image and the second image have different corresponding image acquisition times; a second processing unit configured to determine, based on the first and second images, first information representative of changes in outer contours of the plurality of objects in the first and second images and second information representative of changes in internal textures of the plurality of objects in the first and second images; and a third processing unit configured to determine whether the plurality of objects have been moved based on the first information and the second information.

[0023] An embodiment of the present application includes a processor and a memory adapted to store a computer program operable on the processor, The present invention further provides an electronic device that is used to perform the steps of any of the above methods when the processor runs the computer program.

[0024] An embodiment of the present application further provides a storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of any of the above methods. Effect of the Invention

[0025] The image processing method, device, electronic device, and storage medium according to the embodiments of the present application acquire a first image and a second image, the corresponding image capture times being different, of a target area in which a plurality of objects placed in a stacked manner are present, and determine, based on the first and second images, first information representing a change in the outer contours of the plurality of objects in the first and second images and second information representing a change in the internal texture of the plurality of objects in the first and second images, and determine, based on the first and second information, whether the plurality of objects have been moved. An aspect of the present application is to recognize changes in object position from two perspectives, namely, overall and local, based on images taken at different times in a target area where multiple objects are placed in a stacked manner; in other words, to determine whether multiple objects have been moved based on changes in the outer contours (i.e., overall) in the images of the multiple objects and changes in the internal textures (i.e., local) in the images of the multiple objects. In this way, the state of the objects is monitored using computer vision technology (i.e., processing of the images of the target area) and it is possible to intelligently determine whether the objects have been moved, thereby avoiding the waste of human resources caused by manual inspections; and, compared to manual inspections by humans, by recognizing changes in object position from two perspectives, namely, overall and local, it is possible to recognize subtle object movement events that are difficult for the human eye to notice, and improve the accuracy of the judgment results of whether the objects have been moved. [Brief description of the drawings]

[0026] [Figure 1] 1 is a schematic diagram illustrating the flow of an image processing method according to an embodiment of the present application. [Diagram 2] 2 is a schematic diagram of contour information and texture information of cargo (i.e., an object) according to an application embodiment of the present application; [Diagram 3] FIG. 2 is a schematic diagram of cargo area detection according to an application embodiment of the present application. [Figure 4] FIG. 2 is a schematic diagram of the division of a cargo area according to an application embodiment of the present application. [Diagram 5] FIG. 2 is a schematic diagram of cargo texture detection according to an application embodiment of the present application; [Figure 6]1 is a structural schematic diagram of an image processing device according to an embodiment of the present application; [Figure 7] 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] The present application will now be described in more detail with reference to the drawings and examples.

[0028] In the related art, whether an object has been moved is usually judged by human patrol inspection, for example, by human patrol inspection to watch over cargo in a warehouse, which wastes human resources and is difficult for the human eye to notice some subtle object movement events.

[0029] In light of this, in each embodiment of the present application, for a target area where multiple objects are placed in a stacked manner, changes in object position are recognized from two perspectives, namely, overall and local, based on images taken at different times. In other words, it is determined whether multiple objects have been moved based on the change in the outer contour in the image of the multiple objects (i.e., overall) and the change in the internal texture in the image of the multiple objects (i.e., local). In this way, the state of the objects is monitored using computer vision technology (i.e., processing of the image of the target area), and it is possible to intelligently determine whether the objects have been moved, thereby avoiding the waste of human resources caused by human patrol inspections. Furthermore, compared to human patrol inspections, by recognizing changes in object position from two perspectives, namely, overall and local, it is possible to recognize subtle object movement events that are difficult for the human eye to notice, and improve the accuracy of the judgment result of whether an object has been moved.

[0030] An embodiment of the present application provides an image processing method applied to an electronic device (e.g., a server), as shown in FIG. 1 , the method includes: In step 101, a first image and a second image of a target area are acquired; wherein there are a plurality of objects placed in a stacked manner in the target area, and the corresponding image acquisition times of the first image and the second image are different; In step 102, first information and second information are determined based on the first image and the second image; wherein the first information represents a change in an outer contour of the plurality of objects in the first image and the second image, and the second information represents a change in an inner texture of the plurality of objects in the first image and the second image, In step 103, it is determined whether the plurality of objects have been moved based on the first information and the second information.

[0031] Wherein, the first image corresponds to a first image capture time, and the second image corresponds to a second image capture time. It can be understood that determining whether the plurality of objects have been moved means determining whether the plurality of objects have been moved within a time range from the first image capture time to the second image capture time.

[0032] In actual application, since the plurality of objects are placed in a stacked manner, the plurality of objects are regarded as one integral object, and when an object located at an edge of the plurality of objects is moved, the outer contour of the plurality of objects in the first image and the second image changes, and when an object located inside the plurality of objects is moved, the inner texture of the plurality of objects in the first image and the second image changes, where the inner texture can be understood as information such as the contour and exterior pattern of the object located inside the plurality of objects.

[0033] In step 101, when actually applied, obtaining the first image and the second image of the target area as described above includes: The method may include obtaining a first image and a second image of the target area from an image capture device having a fixed position and image capture angle, the first image and the second image being captured by the image capture device.

[0034] In practical applications, in some business scenarios, there may be multiple objects placed in multiple locations in a stacked manner, such as cargo piled up in separate piles in a warehouse. In order to improve image processing efficiency, the image capture device can capture an image including multiple piles of objects, and the electronic device can obtain an image including multiple piles of objects from the image capture device, detect an area where each pile of objects is located from the image, and perform steps 101 to 103 for each of the detected areas to determine whether or not each pile of objects has a moved object.

[0035] In light of this, in one embodiment, the acquiring of the first image and the second image of the target area as described above includes: acquiring a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image having different corresponding image acquisition times; determining, based on the ninth and tenth images, at least one second region in which a plurality of objects are present in a stacked manner in the first region; determining a target area from the at least one second area; and cropping the ninth and tenth images based on the target area to obtain the first and second images.

[0036] Here, it can be understood that in the first area there is at least one pile of a plurality of objects placed in a stacked manner, and in each second area there is one pile of a plurality of objects placed in a stacked manner.

[0037] In practical application, the electronic device can obtain a ninth image and a tenth image captured by the image capture device from the image capture device, the ninth image corresponds to the first image capture time, and the tenth image corresponds to the second image capture time. In addition, when capturing the ninth and tenth images, the position of the image capture device may be fixed or not, and the image capture angle may be fixed or not, and can be specifically set as needed, and the embodiment of the present application is not limited thereto.

[0038] In practical application, if the position of the image capture device and / or the image capture angle are not fixed, it can be understood that it is necessary to process the 9th image and the 10th image by a method such as image comparison so that the object in the 9th image corresponds to the object in the 10th image.

[0039] In practical application, a pre-trained model can be used to determine each mountain object in the first region, i.e., to determine at least one second region in the first region.

[0040] In view of this, in one embodiment, determining at least one second region in the first region based on the ninth image and the tenth image as described above includes: The method may include determining at least one second region in the first region using the ninth image, the tenth image and a third model trained with a target detection algorithm.

[0041] In practical application, the target detection algorithm may include yolo_v5, faster-rcnn, centerNet, etc., and can be specifically set according to needs, and the embodiments of the present application are not limited thereto.

[0042] In practical application, the third model needs to be pre-trained. Specifically, a training data set may be determined that includes a pre-defined number (e.g., 2000) of images of a pre-defined area (which may be set as necessary, and there must be a plurality of objects stacked in the area) captured by the image capture device, each mountain object in each image is defined, and the coordinate information corresponding to each mountain object is recorded (i.e., labeled). After completing the labeling, the third model can be trained using the labeled data and the target detection algorithm.

[0043] In actual application, when the position of the image collection device and / or the image collection angle are not fixed, it can be understood that it is necessary to input the 9th image into the third model to obtain at least one candidate second region output by the third model, then input the 10th image into the third model to obtain at least one candidate second region output by the third model, and determine at least one second region by associating the at least one candidate second region output by the third model based on the 9th image with the at least one candidate second region output by the third model based on the 10th image (the second region as a candidate corresponding to the same mountain object output by the third model both times may be determined as the second region).

[0044] When the position and image capture angle of the image capture device are fixed, only the ninth image needs to be input to the third model to obtain at least one second region output by the third model.

[0045] In practical application, the second region may be rectangular, and the third model can be used to detect the rectangular region occupied by each mountain object from the image captured by the image capture device, thereby ensuring that information unrelated to external objects does not interfere with the subsequent image processing flow.

[0046] In step 102, if practically applied, a pre-trained model may be used to process the first and second images to determine the first information.

[0047] In light of this, in one embodiment, determining the first information based on the first image and the second image as described above includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image, binarizing the second image using the first model to obtain a fourth image, and setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as a first value, and setting pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects (i.e., pixel points corresponding to non-objects) as a second value; and determining the first information by comparing the third image with the fourth image.

[0048] In practical application, the first value may be 1 and the second value may be 0.

[0049] When actually applied, the semantic segmentation algorithm may include deeplab_v3, U-net, etc., and can be specifically set as needed, and the embodiments of the present application are not limited thereto.

[0050] In practical application, the first model needs to be pre-trained. Specifically, a training data set may be determined, which may include a pre-set number (e.g., 2000) of images (which may be the same as the images used to train the third model) of a pre-set area (which may be set as necessary, and there must be multiple objects stacked in the area) collected by the image collection device, and the coordinate position of the outer contour of each mountain object in each image is marked, and the first model can be trained using the marked data and the semantic segmentation algorithm.

[0051] In one embodiment, determining the first information by comparing the third image with the fourth image includes: determining a plurality of first coefficients based on the third image and the fourth image, each of the first coefficients representing whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; determining the first information from the plurality of first coefficients.

[0052] In practical application, a specific manner for determining the first coefficient can be set according to need. For example, when a pixel value of a pixel point in the third image is the same as a pixel value in the fourth image, the first coefficient may be equal to 0, and when a pixel value of a pixel point in the third image is different from a pixel value in the fourth image, the first coefficient may be equal to 1.

[0053] In one embodiment, determining the first information by the plurality of first coefficients includes: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; The method may include determining whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that an outer contour of the plurality of objects in the first image and the second image has changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that an outer contour of the plurality of objects in the first image and the second image has not changed.

[0054] In practical application, a specific manner for calculating the second coefficient can be set as required. It can be understood that the larger the second coefficient is, the lower the matching degree between the third image and the fourth image is.

[0055] In practical application, the first threshold value may be determined by statistically calculating the effect of the first model on a preset validation data set.

[0056] With this in mind, in one embodiment, the method comprises: determining a first probability, a second probability, a third probability, and a fourth probability of the first model, the first probability representing the probability that the first model recognizes an object in an input image (e.g., the first image and the second image) as an object, i.e., the probability of determining a pixel value of a pixel point corresponding to an object as a first value in a process of binarizing the input image, the second probability representing the probability that the first model recognizes an object in the input image as a non-object, i.e., the probability of determining a pixel value of a pixel point corresponding to an object as a second value in a process of binarizing the input image, the third probability representing the probability that the first model recognizes a non-object in the input image as an object, i.e., the probability of determining a pixel value of a pixel point corresponding to a non-object as a first value in a process of binarizing the input image, and the fourth probability representing the probability that the first model recognizes a non-object in the input image as a non-object, i.e., the probability of determining a pixel value of a pixel point corresponding to a non-object as a second value in a process of binarizing the input image; The method may further include determining the first threshold using the first probability, the second probability, the third probability, and the fourth probability.

[0057] Here, the first probability, the second probability, the third probability and the fourth probability can be determined by statistically analyzing the effect of the first model in a preset validation data set.

[0058] In practical application, a specific manner of determining the first threshold using the first probability, the second probability, the third probability, and the fourth probability can be set as necessary. For example, when using the first probability, the second probability, the third probability, and the fourth probability, the first threshold can be determined by combining with ideas such as Bernoulli distribution, binomial distribution, central limit theorem, Gaussian distribution, and 3σ method.

[0059] In step 102, if practically applied, a pre-trained model can be used to process the first and second images to determine the second information.

[0060] In light of this, in one embodiment, determining the second information based on the first image and the second image as described above includes: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image, binarizing the second image using the second model to obtain a sixth image, and setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; determining the second information from at least the fifth image and the sixth image.

[0061] Here, the first value may be 1 and the second value may be 0.

[0062] In practical application, the edge detection algorithm may include PiDiNet, etc., and can be specifically set according to needs, and the embodiments of the present application are not limited thereto.

[0063] When applied in practice, the second model may be pre-trained, or the second model may be an open source model.

[0064] When actually applied, the second information may be determined by the third image, the fourth image, the fifth image and the sixth image to further ensure that information unrelated to external objects does not interfere with subsequent image processing flow.

[0065] In light of this, in one embodiment, determining the second information by at least the fifth image and the sixth image as described above includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image, binarizing the second image using the first model to obtain a fourth image, setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as a first value, and setting pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as a second value; determining the second information using the third image, the fourth image, the fifth image, and the sixth image.

[0066] Specifically, in one embodiment, determining the second information by the third image, the fourth image, the fifth image, and the sixth image includes: aligning and multiplying the third image and the fifth image to obtain a seventh image, and aligning and multiplying the fourth image and the sixth image to obtain an eighth image; determining the second information by comparing the seventh image with the eighth image.

[0067] Here, by aligning and multiplying the third image and the fifth image, and aligning and multiplying the fourth image and the sixth image, interference from information unrelated to external objects can be eliminated; in other words, the seventh image and the eighth image do not contain information unrelated to external objects; and determining the second information by comparing the seventh image and the eighth image ensures that information unrelated to external objects does not interfere with the subsequent image processing flow, and can further improve the accuracy of the judgment result.

[0068] In actual application, there may be a large number of objects stacked on top of each other. In order to further improve the accuracy of the judgment result, a grid division is first performed on the seventh image and the eighth image, and then the grid of the seventh image is compared with the grid of the eighth image on a grid basis to determine the local texture change situation of the objects stacked on top of each other.

[0069] In light of this, in one embodiment, determining the second information by comparing the seventh image with the eighth image as described above includes: Dividing the seventh image according to a preset rule to obtain a plurality of first grids, and dividing the eighth image according to the preset rule to obtain a plurality of second grids; determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a matching degree between a first lattice and a corresponding second lattice; determining the second information using the plurality of third coefficients.

[0070] Here, the preset rule can be set as necessary. For example, the preset rule can include: partitioning an image into H×W grids, where H and W are integers greater than 0, H and W can be the same or different, and specific values ​​of H and W can be set according to experience.

[0071] Specifically, in one embodiment, determining the second information by the plurality of third coefficients includes: determining whether each third coefficient is greater than a second threshold; When a third coefficient exists that is greater than the second threshold, the second information indicates that the internal texture in the first image and the second image of the plurality of objects has changed, or when each third coefficient is less than or equal to the second threshold, the second information indicates that the internal texture in the first image and the second image of the plurality of objects has not changed.

[0072] In practical application, the second threshold value can be determined by statistically calculating the effect of the second model on a preset validation data set.

[0073] With this in mind, in one embodiment, the method comprises: determining a fifth probability, a sixth probability, a seventh probability, and an eighth probability of the second model, the fifth probability representing the probability that the second model recognizes an edge in an input image (e.g., the first image and the second image) as an edge, i.e., the probability of determining a pixel value of a pixel point corresponding to an edge as a first value in the process of binarizing the input image, the sixth probability representing the probability that the second model recognizes an edge in an input image as a non-edge, i.e., the probability of determining a pixel value of a pixel point corresponding to an edge as a second value in the process of binarizing the input image, the seventh probability representing the probability that the second model recognizes a non-edge in an input image as an edge, i.e., the probability of determining a pixel value of a pixel point corresponding to a non-edge as a first value in the process of binarizing the input image, and the eighth probability representing the probability that the second model recognizes a non-edge in an input image as a non-edge, i.e., the probability of determining a pixel value of a pixel point corresponding to a non-edge as a second value in the process of binarizing the input image; The method may further include determining the second threshold using the fifth probability, the sixth probability, the seventh probability, and the eighth probability.

[0074] Here, the fifth, sixth, seventh and eighth probabilities can be determined by statistically analyzing the effect of the second model in a preset validation data set.

[0075] In practical application, a specific manner of determining the second threshold using the fifth, sixth, seventh, and eighth probabilities can be set as necessary. For example, when using the fifth, sixth, seventh, and eighth probabilities, the second threshold can be determined by combining with ideas such as Bernoulli distribution, binomial distribution, central limit theorem, Gaussian distribution, and 3σ method.

[0076] In one embodiment, in step 103, determining whether the plurality of objects have been moved based on the first information and the second information includes: determining that at least one of the plurality of objects has been moved if the first information indicates that an outer contour of the plurality of objects in the first and second images has changed and / or if the second information indicates that an inner texture of the plurality of objects in the first and second images has changed; Alternatively, the method may include determining that the plurality of objects has not been moved when the first information indicates that an outer contour of the plurality of objects in the first image and the second image has not changed, and the second information indicates that an internal texture of the plurality of objects in the first image and the second image has not changed.

[0077] In practical application, when it is determined that at least one of the multiple objects has been moved, warning information can be issued to the target device to alert the user to the presence of an object that has been moved in the target area.

[0078] With this in mind, in one embodiment, the method comprises: The method may further include issuing a warning when it is determined that at least one object of the plurality of objects has been moved; Wherein, the second information is determined by a plurality of third coefficients each representing a degree of matching between a first grid and a corresponding second grid, and when the first image corresponds to a plurality of first grids and the second image corresponds to a plurality of second grids, and the second information represents that internal textures in the first and second images of the plurality of objects have changed, the warning information includes at least one grid marker for locating a moved object, each corresponding to a third coefficient greater than a second threshold value.

[0079] In practical application, a specific destination (i.e., the target device) of the warning information can be set as necessary, and the embodiment of the present application is not limited thereto.

[0080] In practical application, the image processing method according to the embodiment of the present application can monitor whether an object moved to a specified area (e.g., a first area) exists. The ninth image may be an image of an original state, i.e., the ninth image can reflect the original state of the object (e.g., the state at the time of storage), and the tenth image may be an image of a current state, i.e., the tenth image can reflect the current state of the object. The ninth image can be updated based on the business state (e.g., new cargo has been stored or cargo has been removed) corresponding to the plurality of objects, and the tenth image can be updated periodically or triggered. The periodic update may include the image capture device capturing a tenth image of the first area at a preset period (which can be set as necessary, e.g., n seconds, n is an integer greater than 0) and transmitting the tenth image to the electronic device, and the triggered update may include the electronic device acquiring a tenth image from the image capture device when the electronic device receives a detection command from another device (e.g., a terminal).

[0081] An image processing method according to an embodiment of the present application acquires a first image and a second image, the corresponding image capture times of which are different, of a target area in which a plurality of objects are placed in a stacked manner, and determines, based on the first and second images, first information representing a change in the outer contours of the plurality of objects in the first and second images and second information representing a change in the internal texture of the plurality of objects in the first and second images, and determines whether the plurality of objects have been moved based on the first and second information. An aspect of the present application is to recognize changes in object position from two perspectives, namely, overall and local, based on images taken at different times in a target area where multiple objects are placed in a stacked manner; in other words, to determine whether multiple objects have been moved based on changes in the outer contours (i.e., overall) in the images of the multiple objects and changes in the internal textures (i.e., local) in the images of the multiple objects. In this way, the state of the objects is monitored using computer vision technology (i.e., processing of the images of the target area) and it is possible to intelligently determine whether the objects have been moved, thereby avoiding the waste of human resources caused by manual inspections; and, compared to manual inspections by humans, by recognizing changes in object position from two perspectives, namely, overall and local, it is possible to recognize subtle object movement events that are difficult for the human eye to notice, and improve the accuracy of the judgment results of whether the objects have been moved.

[0082] The present invention will be described in more detail below with reference to application examples.

[0083] This application embodiment provides a warehouse cargo (i.e., the above object) surveillance aspect based on computer vision, and recognizes changes in the cargo from two perspectives: global and local. Among them, the change in the global perspective of the cargo is the change in the outer contour, and the change in the local perspective is the change in some texture of the corresponding area in the cargo image. Cargo is generally stored in a stacked form in a warehouse, and if the cargo at the edge of the cargo area in the image is moved, the contour information at the corresponding position (i.e., the above first information) will be changed, and the contour information is as shown in the square frame 201 in FIG. 2. If the cargo inside the cargo area in the image is moved, the texture information at the corresponding position (i.e., the above second information) will be changed, and the texture information is as shown in the square frame 202 in FIG. 2.

[0084] In this application example, in order to effectively recognize the above phenomenon (i.e., changes in contour information and changes in texture information), the warehouse cargo watch mode is as follows: Step 1 of detecting cargo regions using a region detection model (i.e. the third model described above); Step 2 of partitioning the cargo area using a partition model (i.e. the first model); Step 3 of detecting cargo texture using an edge detection model (i.e. the second model); and step 4 of performing final matching.

[0085] Each of the above four steps complements the others, and the specific implementation of each step will be explained below.

[0086] First, a specific implementation of detecting a cargo region using a region detection model (i.e., the above third model) will be described.

[0087] In this application embodiment, the role of the region detection model is to detect the rectangular region (i.e., the second region) occupied by the cargo from the images (e.g., the ninth and tenth images) captured by the monitoring camera head (i.e., the image capture device) to ensure that external information unrelated to the cargo does not interfere in the subsequent flow. The specific effect is to define each pile of cargo and obtain a corresponding rectangular frame (e.g., rectangular frame 301) as shown in Figure 3.

[0088] In this application example, the region detection model uses the yolo_v5 algorithm to detect regions. When training the region detection model, about 2000 pictures detected by the camera head in the warehouse are selected, and all the piles of cargo in each picture are defined, and the coordinate information corresponding to each rectangular frame is recorded. After the labeling is completed, the region detection model is trained by the labeled data. The final region detection model can detect each pile of cargo in the newly captured pictures and define it with the corresponding rectangular frame.

[0089] Next, a specific implementation of dividing the cargo area according to the division model (i.e., the above first model) will be described.

[0090] In this application example, the role of the segmentation model is to obtain the outer contour of each pile of cargo and judge whether the cargo contour information has changed. The segmentation model processes the image using the deeplab_v3 algorithm, and the output of the segmentation model (as shown in Figure 4) is a matrix (e.g., the third and fourth images) whose size is the same as the input picture, in which the pixel position corresponding to the cargo in the matrix has a value of 1 (i.e., the first value) and the other parts (i.e., non-cargo parts) have a value of 0 (i.e., the second value).

[0091] In this application example, it is necessary to obtain the marked data in advance and train the segmentation model by the marked data. The method of obtaining the marked data may include: selecting about 2000 pictures (the pictures that have been used in training the region detection model can be used) detected by the camera head in the warehouse, marking the coordinate position of the outer contour of each pile of cargo in the pictures, and training the segmentation model by the marks. After the training is completed, the segmentation model analyzes the new input picture and generates a 0,1 matrix whose size is the same as the input picture, in which the area corresponding to the cargo is 1 and the other parts are 0.

[0092] Thirdly, a concrete implementation of detecting cargo texture using an edge detection model (i.e., the above second model) will be described.

[0093] In this application example, the role of the edge detection model is to recognize local textures in the input picture (e.g., the first and second images) and determine whether the texture information has changed from the original. The edge detection model uses PiDiNet to extract important textures by recognizing areas of sudden changes inside the picture, and the image output by the edge detection model is a 0, 1 matrix with one size similar to the original picture, as shown in Figure 5, where the value of the pixel position corresponding to the edge is 1 (i.e., the first value above) and the values ​​of other positions are 0 (i.e., the second value above).

[0094] In practical applications, the edge detection model may be an open source model and does not need to be retrained, since edge detection is a general-purpose algorithm and is not limited to cargo edge detection.

[0095] Fourth, we provide a detailed explanation of the final matching process.

[0096] In this application embodiment, the three steps of cargo area detection, cargo area division, and cargo texture detection all serve the final recognition (i.e., final matching) process. After the cargo enters the warehouse, an original picture of the cargo needs to be obtained by the camera head for monitoring the cargo, and then subsequent pictures of the cargo need to be periodically taken. Each time a subsequent picture is obtained, it needs to be compared with the original picture to determine whether the cargo has been moved. The original picture of the cargo and the subsequently obtained pictures all need to go through the three steps of cargo area detection, cargo area division, and cargo texture detection before the final comparison.

[0097] In practical application, in a warehouse scene, the position and angle of the surveillance camera head are usually constant. Therefore, the cargo area detection can be performed only on the original cargo picture, and the obtained detection frame is also used for the subsequently acquired cargo picture at the same time. However, both the original cargo picture and the subsequently acquired cargo picture need to undergo cargo area segmentation and cargo texture detection.

[0098] In an actual application, when the cargo status is updated, for example, when new cargo is received or discharged, the original picture can be updated.

[0099] In this application example, after completing the above steps (i.e. cargo area detection, cargo area segmentation, and cargo texture detection), the original cargo picture (

number

number

number

number

number

number

number

[0100] 1) Subpicture

number

number

number

number

[0101] 2) Calculate the coefficient of variation (i.e., the second coefficient above) using the following formula:

number

number

number

number

number

number

number

number

[0102] 3) If the coefficient of variation is greater than a certain threshold (ie, the above-mentioned first threshold), it is determined that one load of cargo corresponding to the subpicture has moved.

[0103] The comparison process for texture information includes the following steps.

[0104] 1) Subpicture

number

number

number

number

[0105] 2) Subpicture

number

number

number

number

[0106] 3)

number

number

number

number

number

number

[0107] 4)

number

number

number

number

[0108] 5) Calculate the coefficient of variation (i.e., the third coefficient above) for each mesh part using the following formula:

number

number

[0109] The reason for dividing the area into meshes and then calculating the coefficient of variation is that movement may occur only in a certain local area, and when comparing the entire picture directly, the difference obtained is small, making it difficult to accurately determine whether cargo has been moved or not, but dividing the area into meshes can effectively solve this problem. Also, dividing the area into meshes can help locate the specific location where movement has occurred.

[0110] 6) If the coefficient of variation is greater than a certain threshold (ie, the above-mentioned second threshold), it is determined that the corresponding part (ie, region) in the corresponding mesh has moved.

[0111] From the above steps, it can be seen that both contour and texture contrast require comparing the coefficient of variation with a corresponding threshold value. However, setting the threshold value is usually a difficult problem. Therefore, in this application embodiment, the threshold value is derived by statistical means to help make the judgment more accurate.

[0112] In this application embodiment, when comparing edge information, the derivation process for the threshold value (ie, the first threshold value) is as follows. We obtain the probability that the segmentation model recognizes the cargo part as the cargo part, the probability that the cargo part is recognized as the other part, the probability that the other part is recognized as the cargo part, and the probability that the other part is recognized as the other part, respectively.

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0113] In this application embodiment, when comparing texture information, the derivation process for the threshold (ie, the second threshold) is as follows: We obtain the probability that the edge detection model recognizes an edge as an edge, the probability that it recognizes an edge as a non-edge, the probability that it recognizes a non-edge as an edge, and the probability that it recognizes a non-edge as a non-edge, respectively.

number

number

number

number

number

number

number

number

number

number

number

number

number

[0114] In this application example, the specific flow of the image processing process is as follows.

[0115] 1) Train the model. Label the data according to the preset rules, train the region detection model and the segmentation model, and obtain the edge detection model. At the same time, the probability values ​​corresponding to the segmentation model and the edge detection model in the validation data set are obtained.

number

number

[0116] 2) Detecting the cargo area: After completing the storage of the cargo or the status update, the corresponding camera head takes the original picture of the cargo, and the detection model is used to obtain the detection frame corresponding to each pile of cargo.

[0117] 3) Segment the cargo area: Cargo area segmentation is performed on the original cargo picture to obtain a corresponding segmentation diagram, and this operation only needs to be performed once when the cargo state is not updated; cargo area segmentation is performed on the periodically captured cargo picture to obtain a corresponding segmentation diagram, and this needs to be performed once every time a new picture is captured.

[0118] 4) Detect cargo texture: perform texture detection through edge detection algorithm on original cargo picture to obtain corresponding edge picture, this operation only needs to be performed once when cargo state is not updated, perform edge detection on periodically captured cargo picture to obtain corresponding edge picture, this needs to be performed once every time a new picture is captured.

[0119] 5) Contrast the contours. Subpictures of each pair

number

number

number

number

[0120] 6) Contrast textures. Subpictures of each pair

number

number

number

[0121] An aspect of this application embodiment is to use computer vision technology to monitor cargo in the warehouse using surveillance camera heads in the warehouse to determine whether it has been moved, and to use image segmentation and edge detection technology to obtain the contour and texture information of the cargo, and based on these two types of information, compare the state (i.e., position) of the cargo to determine whether it has been moved from two perspectives, namely, global and local, and further derive and obtain a threshold value of the difference between the original state of the cargo and the later collected state using statistical methods, which helps to more accurately determine whether the cargo has been moved.

[0122] The embodiment of this application realizes monitoring of the state (i.e., location) of warehouse cargo based on computer vision technology, and when a change in the state (i.e., location) of the cargo is detected, an alarm is issued without delay and a request for human inspection is made. Since there are generally a large number of surveillance camera heads in warehouses, such a method can fully utilize traditional resources and effectively reduce manpower consumption. At the same time, computer vision can also recognize some subtle changes that are difficult for the human eye to notice.

[0123] To realize the method of the embodiment of the present application, the embodiment of the present application further provides an image processing device provided in an electronic device (for example, attached to a server), as shown in FIG. 6 , the device includes: a first processing unit 601 configured to acquire a first image and a second image of a target area in which a plurality of objects are present, the first image and the second image have different corresponding image acquisition times; a second processing unit 602 configured to determine, based on the first and second images, first information representative of changes in outer contours of the objects in the first and second images and second information representative of changes in internal textures of the objects in the first and second images; and a third processing unit 603 configured to determine whether the plurality of objects have been moved based on the first information and the second information.

[0124] Wherein, in one embodiment, the second processing unit 602 further comprises: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values ​​and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; The first information is determined by comparing the third image with the fourth image.

[0125] In one embodiment, the second processing unit 602 further comprises: determining a plurality of first coefficients based on the third image and the fourth image, each of which indicates whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; The plurality of first coefficients are configured to determine the first information.

[0126] In one embodiment, the second processing unit 602 further comprises: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; The method is configured to determine whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that the outer contours of the multiple objects in the first and second images have changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that the outer contours of the multiple objects in the first and second images have not changed.

[0127] In one embodiment, the second processing unit 602 further comprises: configured to determine a first probability, a second probability, a third probability, and a fourth probability of the first model, and to determine the first threshold using the first probability, the second probability, the third probability, and the fourth probability; The first probability represents the probability that the first model recognizes an object in the input image as an object, the second probability represents the probability that the first model recognizes an object in the input image as a non-object, the third probability represents the probability that the first model recognizes a non-object in the input image as an object, and the fourth probability represents the probability that the first model recognizes a non-object in the input image as a non-object.

[0128] In one embodiment, the second processing unit 602 further comprises: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image; binarizing the second image using the second model to obtain a sixth image; and setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; At least the fifth image and the sixth image are configured to determine the second information.

[0129] In one embodiment, the second processing unit 602 further comprises: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values ​​and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; The third image, the fourth image, the fifth image and the sixth image are configured to determine the second information.

[0130] In one embodiment, the second processing unit 602 further comprises: aligning and multiplying the third image and the fifth image to obtain a seventh image; aligning and multiplying the fourth image and the sixth image to obtain an eighth image; The seventh image is compared with the eighth image to determine the second information.

[0131] In one embodiment, the second processing unit 602 further comprises: Dividing the seventh image according to a preset rule to obtain a plurality of first grids; dividing the eighth image according to the preset rule to obtain a plurality of second grids; determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a matching degree between a first lattice and a corresponding second lattice; The plurality of third coefficients are configured to determine the second information.

[0132] In one embodiment, the second processing unit 602 further comprises: determining whether each third coefficient is greater than a second threshold; The second information is configured such that if there is a third coefficient greater than the second threshold, the second information indicates that the internal texture in the first and second images of the plurality of objects has changed, or if each third coefficient is less than or equal to the second threshold, the second information indicates that the internal texture in the first and second images of the plurality of objects has not changed.

[0133] In one embodiment, the second processing unit 602 further comprises: determining a fifth probability, a sixth probability, a seventh probability, and an eighth probability of the second model; configured to determine the second threshold using the fifth, sixth, seventh, and eighth probabilities; The fifth probability represents the probability that the second model recognizes an edge in the input image as an edge, the sixth probability represents the probability that the second model recognizes an edge in the input image as a non-edge, the seventh probability represents the probability that the second model recognizes a non-edge in the input image as an edge, and the eighth probability represents the probability that the second model recognizes a non-edge in the input image as a non-edge.

[0134] In one embodiment, the third processing unit 603 further comprises: determining that at least one of the plurality of objects has been moved if the first information indicates that an outer contour of the plurality of objects in the first image and the second image has changed and / or if the second information indicates that an inner texture of the plurality of objects in the first image and the second image has changed; Alternatively, the method is configured to determine that the plurality of objects have not been moved when the first information indicates that the outer contours of the plurality of objects in the first and second images have not changed and the second information indicates that the internal textures of the plurality of objects in the first and second images have not changed.

[0135] In one embodiment, the device further comprises a communication unit, and the third processing unit 603 is further configured to issue warning information through the communication module when determining that at least one object among the plurality of objects has been moved, wherein: The second information is determined by a plurality of third coefficients each representing a degree of matching between one first grid and a corresponding second grid, and when the first image corresponds to a plurality of first grids and the second image corresponds to a plurality of second grids, and the second information represents that internal textures in the first and second images of the plurality of objects have changed, the warning information includes at least one grid marker for locating a moved object, each corresponding to a third coefficient greater than a second threshold.

[0136] In one embodiment, the first processing unit 601 further comprises: Acquire a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image having different corresponding image acquisition times; determining at least one second region in the first region where a plurality of objects are present in a stacked manner based on the ninth image and the tenth image; The apparatus is configured to determine a target area from the at least one second area, and crop the ninth and tenth images based on the target area to obtain the first and second images.

[0137] In one embodiment, the first processing unit 601 is further configured to determine at least one second region in the first region using the ninth image, the tenth image and a third model trained by a target detection algorithm.

[0138] In practical application, the communication unit can be realized by a communication interface in an image processing device, and the first processing unit 601, the second processing unit 602 and the third processing unit 603 can be realized by a processor in the image processing device.

[0139] In addition, the image processing device according to the above embodiment has been described using only the partitions of the above program modules as an example when processing an image, but in actual application, the above processes can be assigned to be completed by different program modules as necessary, that is, the internal structure of the device can be partitioned into different program modules to complete all or part of the processes described above. In addition, the image processing device according to the above embodiment belongs to the same concept as the embodiment of the image processing method, and the details of the specific realization process thereof are to be referred to the embodiment of the method, and will not be described again here.

[0140] In order to realize the above program modules based on hardware and to realize the method of the embodiment of the present application, the embodiment of the present application further provides an electronic device, as shown in FIG. 7, the electronic device 700 includes: A communication interface 701 capable of performing information interaction with other electronic devices; A processor 702 connected to the communication interface 701 to realize information interaction with other electronic devices, and used to execute the method according to one or more of the above technical aspects when a computer program is running; and a memory 703 for storing a computer program operable on the processor 702.

[0141] Specifically, the processor 702: acquiring a first image and a second image of a target area in which a plurality of objects are placed in a stacked manner, the first image and the second image being taken at different times; determining, based on the first and second images, first information representing a change in an outer contour of the plurality of objects in the first and second images and second information representing a change in an inner texture of the plurality of objects in the first and second images; The system is configured to determine whether the plurality of objects have been moved based on the first information and the second information.

[0142] Wherein, in one embodiment, the processor 702 further comprises: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values ​​and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; The first information is determined by comparing the third image with the fourth image.

[0143] In one embodiment, the processor 702 further comprises: determining a plurality of first coefficients based on the third image and the fourth image, each of which indicates whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; The plurality of first coefficients are configured to determine the first information.

[0144] In one embodiment, the processor 702 further comprises: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; The method is configured to determine whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that the outer contours of the multiple objects in the first and second images have changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that the outer contours of the multiple objects in the first and second images have not changed.

[0145] In one embodiment, the processor 702 further comprises: configured to determine a first probability, a second probability, a third probability, and a fourth probability of the first model, and to determine the first threshold using the first probability, the second probability, the third probability, and the fourth probability; The first probability represents the probability that the first model recognizes an object in the input image as an object, the second probability represents the probability that the first model recognizes an object in the input image as a non-object, the third probability represents the probability that the first model recognizes a non-object in the input image as an object, and the fourth probability represents the probability that the first model recognizes a non-object in the input image as a non-object.

[0146] In one embodiment, the processor 702 further comprises: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image; binarizing the second image using the second model to obtain a sixth image; and setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; At least the fifth image and the sixth image are configured to determine the second information.

[0147] In one embodiment, the processor 702 further comprises: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image; binarizing the second image using the first model to obtain a fourth image; setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image as first values ​​and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects as second values; The third image, the fourth image, the fifth image and the sixth image are configured to determine the second information.

[0148] In one embodiment, the processor 702 further comprises: aligning and multiplying the third image and the fifth image to obtain a seventh image; aligning and multiplying the fourth image and the sixth image to obtain an eighth image; The seventh image is compared with the eighth image to determine the second information.

[0149] In one embodiment, the processor 702 further comprises: Dividing the seventh image according to a preset rule to obtain a plurality of first grids; dividing the eighth image according to the preset rule to obtain a plurality of second grids; determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a matching degree between a first lattice and a corresponding second lattice; The plurality of third coefficients are configured to determine the second information.

[0150] In one embodiment, the processor 702 further comprises: determining whether each third coefficient is greater than a second threshold; The second information is configured such that if there is a third coefficient greater than the second threshold, the second information indicates that the internal texture in the first and second images of the plurality of objects has changed, or if each third coefficient is less than or equal to the second threshold, the second information indicates that the internal texture in the first and second images of the plurality of objects has not changed.

[0151] In one embodiment, the processor 702 further comprises: determining a fifth probability, a sixth probability, a seventh probability, and an eighth probability of the second model; configured to determine the second threshold using the fifth, sixth, seventh, and eighth probabilities; The fifth probability represents the probability that the second model recognizes an edge in the input image as an edge, the sixth probability represents the probability that the second model recognizes an edge in the input image as a non-edge, the seventh probability represents the probability that the second model recognizes a non-edge in the input image as an edge, and the eighth probability represents the probability that the second model recognizes a non-edge in the input image as a non-edge.

[0152] In one embodiment, the processor 702 further comprises: determining that at least one of the plurality of objects has been moved if the first information indicates that an outer contour of the plurality of objects in the first image and the second image has changed and / or if the second information indicates that an inner texture of the plurality of objects in the first image and the second image has changed; Alternatively, the method is configured to determine that the plurality of objects have not been moved when the first information indicates that the outer contours of the plurality of objects in the first and second images have not changed and the second information indicates that the internal textures of the plurality of objects in the first and second images have not changed.

[0153] In one embodiment, the processor 702 is further configured to issue an alert via the communication interface 701 when determining that at least one object of the plurality of objects has been moved, wherein: The second information is determined by a plurality of third coefficients each representing a degree of matching between one first grid and a corresponding second grid, and when the first image corresponds to a plurality of first grids and the second image corresponds to a plurality of second grids, and the second information represents that internal textures in the first and second images of the plurality of objects have changed, the warning information includes at least one grid marker for locating a moved object, each corresponding to a third coefficient greater than a second threshold.

[0154] In one embodiment, the processor 702 further comprises: Acquire a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image having different corresponding image acquisition times; determining at least one second region in the first region where a plurality of objects are present in a stacked manner based on the ninth image and the tenth image; The apparatus is configured to determine a target area from the at least one second area, and crop the ninth and tenth images based on the target area to obtain the first and second images.

[0155] In one embodiment, the processor 702 is further configured to determine at least one second region in the first region using the ninth image, the tenth image and a third model trained with a target detection algorithm.

[0156] It should be noted that for details of the process in which the processor 702 specifically performs the above operations, please refer to the method embodiments and will not be described again here.

[0157] Of course, in practical applications, the components in the electronic device 700 are coupled together via a bus system 704. It can be understood that the bus system 704 is used to realize connection communication between these components. The bus system 704 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, various buses are labeled as the bus system 704 in FIG. 7.

[0158] Memory 703 in the present embodiment is used to store various types of data to support the operation of electronic device 700. Examples of such data include any computer programs intended to be operated on electronic device 700.

[0159] The methods disclosed in the above embodiments of the present application may be applied to or realized by the processor 702. The processor 702 may be an integrated circuit chip having a signal processing capability. In the process of realization, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 702, or by instructions in the form of software. The above processor 702 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, etc. The processor 702 may realize or execute each method, step, and logic block diagram disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor, or any ordinary processor, etc. The steps of the methods disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decode processor, or a combination of hardware and software modules of a decode processor. The software modules may be located in a storage medium. The storage medium is located in memory 703, and processor 702 reads the information in memory 703 and completes the steps of the above-mentioned method in combination with its hardware.

[0160] In an exemplary embodiment, electronic device 700 may be implemented with one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), general-purpose processors, controllers, Micro Controller Units (MCUs), Microprocessors, or other electronic components to perform the methods.

[0161] It is understood that the memory 703 of the embodiment of the present application may be a volatile memory or a non-volatile memory, and may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a read-only disk (CD-ROM), and the magnetic surface memory may be a magnetic disk memory or a magnetic tape memory. The volatile memory may be a random access memory (RAM) used as an external cache.By way of example and not of limitation, many types of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), SyncLink dynamic random access memory (SLDRAM), and direct Rambus random access memory (DRRAM). The memory described in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

[0162] In an exemplary embodiment, the embodiment of the present application further provides a storage medium, i.e., a computer storage medium, which is specifically a computer readable storage medium, such as a memory 703 for storing a computer program, which is executable by the processor 702 of the electronic device 700 to complete the steps of the above-mentioned method. The computer readable storage medium may be a memory, such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM.

[0163] It should be noted that "first," "second," etc. need not be used to describe a particular order or chronology, but are intended to distinguish between similar objects.

[0164] Furthermore, the technical aspects described in the embodiments of the present application can be combined in any manner as long as they do not conflict with one another.

[0165] The above are only preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application.

Claims

1. acquiring a first image and a second image of a target area in which a plurality of objects are present, the first image and the second image being captured at different corresponding image capture times; determining, based on the first and second images, first information representing a change in an outer contour of the plurality of objects in the first and second images and second information representing a change in an inner texture of the plurality of objects in the first and second images; determining whether the plurality of objects have been moved based on the first information and the second information. Image processing methods.

2. The step of determining the first information based on the first image and the second image includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image, binarizing the second image using the first model to obtain a fourth image, and setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image to a first value and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects to a second value; and determining the first information by comparing the third image with the fourth image. The image processing method according to claim 1 .

3. The step of determining the first information by comparing the third image with the fourth image includes the steps of: determining a plurality of first coefficients based on the third image and the fourth image, each coefficient representing whether a pixel value in the third image of a pixel point is the same as a pixel value in the fourth image; determining the first information by the plurality of first coefficients; The image processing method according to claim 2 .

4. The step of determining the first information by the plurality of first coefficients includes the steps of: determining a second coefficient representing a degree of matching between the third image and the fourth image based on the plurality of first coefficients; determining whether the second coefficient is greater than a first threshold, and if the second coefficient is greater than the first threshold, the first information represents that an outer contour of the plurality of objects in the first image and the second image has changed, or if the second coefficient is less than or equal to the first threshold, the first information represents that an outer contour of the plurality of objects in the first image and the second image has not changed. The image processing method according to claim 3.

5. determining a first probability, a second probability, a third probability and a fourth probability of the first model, the first probability representing a probability that the first model will recognize an object in the input image as an object, the second probability representing a probability that the first model will recognize an object in the input image as a non-object, the third probability representing a probability that the first model will recognize a non-object in the input image as an object, and the fourth probability representing a probability that the first model will recognize a non-object in the input image as a non-object; determining the first threshold using the first probability, the second probability, the third probability, and the fourth probability. The image processing method according to claim 4.

6. The step of determining second information based on the first image and the second image includes: binarizing the first image using a second model trained by an edge detection algorithm to obtain a fifth image, binarizing the second image using the second model to obtain a sixth image, and setting pixel values ​​of pixel points corresponding to edges in the fifth image and the sixth image to a first value and pixel values ​​of pixel points corresponding to non-edges to a second value; determining the second information by at least the fifth image and the sixth image; The image processing method according to claim 1 .

7. The step of determining the second information by at least the fifth image and the sixth image includes: binarizing the first image using a first model trained by a semantic segmentation algorithm to obtain a third image, binarizing the second image using the first model to obtain a fourth image, and setting pixel values ​​of pixel points corresponding to the plurality of objects in the third image and the fourth image to a first value and pixel values ​​of pixel points other than the pixel points corresponding to the plurality of objects to a second value; determining the second information by the third image, the fourth image, the fifth image, and the sixth image; The image processing method according to claim 6.

8. The step of determining the second information by the third image, the fourth image, the fifth image, and the sixth image includes the steps of: aligning and multiplying the third image and the fifth image to obtain a seventh image, and aligning and multiplying the fourth image and the sixth image to obtain an eighth image; and determining the second information by comparing the seventh image with the eighth image. The image processing method according to claim 7.

9. The step of determining the second information by comparing the seventh image with the eighth image includes the steps of: partitioning the seventh image based on a preset rule to obtain a plurality of first grids, and partitioning the eighth image based on the preset rule to obtain a plurality of second grids; determining a plurality of third coefficients based on the plurality of first lattices and the plurality of second lattices, each of the third coefficients representing a degree of matching between one of the first lattices and a corresponding one of the second lattices; determining the second information by the plurality of third coefficients; The image processing method according to claim 8.

10. The step of determining the second information by the plurality of third coefficients includes the steps of: determining whether each third coefficient is greater than a second threshold; When there is a third coefficient greater than the second threshold, the second information represents that an internal texture in the first image and the second image of the plurality of objects has changed, or when each of the third coefficients is less than or equal to the second threshold, the second information represents that an internal texture in the first image and the second image of the plurality of objects has not changed. The image processing method according to claim 9.

11. determining a fifth probability, a sixth probability, a seventh probability and an eighth probability of the second model, the fifth probability representing the probability that the second model will recognize an edge in the input image as an edge, the sixth probability representing the probability that the second model will recognize an edge in the input image as a non-edge, the seventh probability representing the probability that the second model will recognize a non-edge in the input image as an edge, and the eighth probability representing the probability that the second model will recognize a non-edge in the input image as a non-edge; determining the second threshold using the fifth, sixth, seventh, and eighth probabilities. The image processing method according to claim 10.

12. The step of determining whether the plurality of objects have been moved based on the first information and the second information includes: determining that at least one of the plurality of objects has been moved if the first information indicates that an outer contour of the plurality of objects in the first and second images has changed and / or if the second information indicates that an inner texture of the plurality of objects in the first and second images has changed; or determining that the plurality of objects has not been moved if the first information indicates that an outer contour of the plurality of objects in the first image and the second image has not changed and the second information indicates that an inner texture of the plurality of objects in the first image and the second image has not changed. The image processing method according to any one of claims 1 to 11.

13. and issuing a warning when it is determined that at least one object of the plurality of objects has been moved; the second information is determined by a plurality of third coefficients each representing a degree of matching between one first grid and a corresponding second grid, the first image corresponds to a plurality of first grids, the second image corresponds to a plurality of second grids, and when the second information represents that an internal texture in the first image and the second image of the plurality of objects has changed, the warning information includes at least one grid mark for locating the moved object, each of which corresponds to a third coefficient greater than a second threshold value; The image processing method according to claim 12.

14. The acquiring of the first image and the second image of the target area includes: acquiring a ninth image and a tenth image of a first area including at least the target area, the ninth image and the tenth image having different corresponding image acquisition times; determining, based on the ninth and tenth images, at least one second region in which a plurality of objects are present in a stacked manner in the first region; determining a target area from the at least one second area; and cropping the ninth and tenth images based on the target area to obtain the first and second images. The image processing method according to any one of claims 1 to 11.

15. The step of determining at least one second region in the first region based on the ninth image and the tenth image includes: determining at least one second region in the first region using the ninth image, the tenth image, and a third model trained with a target detection algorithm; The image processing method according to claim 14.

16. a first processing unit configured to acquire a first image and a second image of a target area in which a plurality of objects are present, the first image and the second image have different corresponding image acquisition times; a second processing unit configured to determine, based on the first and second images, first information representative of changes in outer contours of the plurality of objects in the first and second images and second information representative of changes in internal textures of the plurality of objects in the first and second images; and a third processing unit configured to determine whether the plurality of objects have been moved based on the first information and the second information. Image processing device.

17. A processor and a memory used to store a computer program operable on the processor, The processor, when running the computer program, is used to carry out the steps of the method according to any one of claims 1 to 15. electronic equipment.

18. A computer program is stored which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 15. storage medium.

Images