Method and apparatus for determining object information, electronic device, and storage medium

Abstract

The present disclosure provides a method and device for determining object information, electronic equipment and storage medium, relates to the field of artificial intelligence technology, in particular to the field of image processing. The specific implementation scheme is: determining at least one image pair according to at least two target images, wherein the image pair includes a first image and a second image, and the first image and the second image have an overlapping part representing the same position; for each image pair, performing target detection on the first image and the second image respectively to obtain a first object detection box corresponding to the first image and a second object detection box corresponding to the second image; and performing deduplication on the object detection boxes in the first object detection box and the second object detection box that indicate the same object to obtain a third object detection box corresponding to each image pair; and determining object information related to the target image according to the third object detection box.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of image processing. More specifically, the present disclosure provides a method, device, electronic device, storage medium, and computer program product for determining object information. Background Art

[0002] In the fast-moving consumer goods industry, the goods on the shelves can be counted to obtain information such as the types and quantities of goods. Then, based on this information, the sales volume, purchase volume and other information of various goods can be determined. Summary of the Invention

[0003] The present disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for determining object information.

[0004] According to one aspect of the present disclosure, a method for determining object information is provided, comprising: determining at least one image pair based on at least two target images, wherein the image pair comprises: a first image and a second image, and the first image and the second image have an overlapping portion representing the same position; performing target detection on the first image and the second image for each image pair, respectively, to obtain a first object detection frame corresponding to the first image and a second object detection frame corresponding to the second image; and deduplicating object detection frames indicating the same object in the first object detection frame and the second object detection frame, to obtain a third object detection frame corresponding to each image pair; and determining object information related to the target image based on the third object detection frame.

[0005] According to another aspect of the present disclosure, a device for determining object information is provided, comprising: an image pair determination module for determining at least one image pair based on at least two target images, wherein the image pair comprises: a first image and a second image, and the first image and the second image have an overlapping portion representing the same position. A deduplication module for performing target detection on the first image and the second image, respectively, for each image pair, to obtain a first object detection frame corresponding to the first image and a second object detection frame corresponding to the second image; deduplication of object detection frames indicating the same object in the first object detection frame and the second object detection frame to obtain a third object detection frame corresponding to each image pair. An information determination module for determining object information related to the target image based on the third object detection frame.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided by the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided in the present disclosure when executed by a processor.

[0009] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an application scenario of the method and apparatus for determining object information according to an embodiment of the present disclosure;

[0012] Figure 2 is a schematic flowchart of a method for determining object information according to an embodiment of the present disclosure;

[0013] Figure 3A is a schematic diagram of a candidate movement trajectory according to an embodiment of the present disclosure;

[0014] Figure 3B is a schematic diagram of another candidate movement trajectory according to an embodiment of the present disclosure;

[0015] Figure 3C is a schematic diagram of another candidate movement trajectory according to an embodiment of the present disclosure;

[0016] Figure 3D is a schematic diagram of another candidate movement trajectory according to an embodiment of the present disclosure;

[0017] Figure 4 is a schematic flow chart of acquiring multiple images according to an embodiment of the present disclosure;

[0018] Figure 5A is a schematic diagram of splicing two images arranged one above the other according to an embodiment of the present disclosure;

[0019] Figure 5B is a schematic diagram of splicing two images arranged left and right according to an embodiment of the present disclosure;

[0020] Figure 5C is a schematic diagram of splicing two columns of images arranged left and right according to an embodiment of the present disclosure;

[0021] Figure 6A is a schematic diagram of a deduplication operation according to an embodiment of the present disclosure;

[0022] Figure 6B is a schematic diagram of a deduplication operation according to an embodiment of the present disclosure;

[0023] Figure 7 is a schematic structural block diagram of an apparatus for determining object information according to an embodiment of the present disclosure; and

[0024] Figure 8 It is a structural block diagram of an electronic device used to implement the method for determining object information according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] With the development of image recognition capabilities, more and more companies are using object recognition technology to identify and count products in a single image. However, when shelves are long and the distance between shelves is narrow, it is difficult to capture the entire shelf in a single image.

[0027] In some technical solutions, multiple images of a target area (e.g., a shelf with goods) can be captured multiple times to obtain multiple target images. These multiple target images can reflect the product information placed in multiple sub-areas of the shelf, and the acquisition perspectives of the multiple target images can be different. A homography matrix can be used to describe the mapping relationship between points on the same plane between different images, and then the multiple target images can be spliced ​​into a complete image based on the homography matrix. Based on the spliced ​​complete image, the product information on the shelf can be counted.

[0028] It should be noted that stitching multiple target images into a complete image has the problem of poor stitching quality. Since product information needs to be counted based on the stitched complete image, the stitching quality of multiple target images will directly affect the accuracy of the statistical results.

[0029] The embodiments of the present disclosure aim to propose a method for determining object information. This method does not require splicing the target image into a complete image. Instead, it obtains a first object detection frame and a second object detection frame included in the target image through target detection, removes the first object detection frame and the second object detection frame, and then counts the object information based on the deduplicated third object detection frame, thereby improving the accuracy of the statistical results.

[0030] The technical solutions provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Figure 1 3 is a schematic diagram of an application scenario of the method and apparatus for determining object information according to an embodiment of the present disclosure.

[0032] It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure may not be used in other devices, systems, environments or scenarios.

[0033] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers, etc.

[0035] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (such as an image obtained by stitching adjacent images, object information determined based on a target image, etc.) to the terminal device.

[0036] It should be noted that the method for determining object information provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the apparatus for determining object information provided in the embodiment of the present disclosure can generally be set in the server 105. The method for determining object information provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the apparatus for determining object information provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0038] Figure 2 is a schematic flowchart of a method for determining object information according to an embodiment of the present disclosure.

[0039] like Figure 2 As shown, the method 200 for determining object information may include operations S210 to S230.

[0040] In operation S210 , at least one image pair is determined based on at least two target images, wherein the image pair includes a first image and a second image, and the first image and the second image have an overlapping portion representing a same position.

[0041] In some embodiments, a video capture device such as a video recorder may be used to capture video data for the target area, and then video frames corresponding to at least two sub-areas in the target area are captured from the video data as target images.

[0042] For example, the target area may include shelves with objects placed on them, such as merchandise, workpieces, or express parcels. The sub-areas may be arranged along a predetermined direction, which may include at least one of a horizontal and a vertical direction. It should be noted that the sub-areas may vary in size and shape.

[0043] In other embodiments, an image acquisition device such as a camera may be used to acquire images of at least two sub-areas in the target area to obtain a target image.

[0044] For example, images of at least two sub-regions within the target area can be captured based on the target movement trajectory, thereby obtaining at least two target images. The target movement trajectory is described in detail below and is not further described here. In other examples, images can be captured without following the target movement trajectory. For example, the target area can be pre-divided into at least two sub-regions, with adjacent sub-regions having overlapping areas, and then images of the sub-regions can be randomly captured.

[0045] For example, when the sub-regions are arranged in a horizontal direction, two target images obtained by capturing images of two adjacent sub-regions located in the same row may be determined as an image pair.

[0046] For another example, when the sub-regions are arranged in a vertical direction, two target images obtained by capturing images of two adjacent sub-regions located in the same column can be determined as an image pair.

[0047] For another example, during the image acquisition process, if it is required that two adjacent images acquired have an overlapping rate, then the two adjacent images based on the acquisition order may be determined as an image pair.

[0048] The overlapping portion may represent a partial image of the same physical area in the first image and the second image. For example, the partial image may represent the same area in a shelf, the same product on the shelf, and the like.

[0049] It should be noted that different image pairs may include the same target image. For example, one image pair includes image pic1 and image pic2, and another image pair includes image pic2 and image pic3.

[0050] In operation S220, for each image pair, object detection is performed on the first image and the second image, respectively, to obtain a first object detection frame corresponding to the first image and a second object detection frame corresponding to the second image. Object detection frames indicating the same object in the first and second object detection frames are deduplicated to obtain a third object detection frame corresponding to each image pair.

[0051] For example, an object detection model can be used to detect objects in the first image and the second image, respectively, to obtain a first object detection frame and a second object detection frame. The object detection model can be a YOLO (You Only Look Once) model. The number of each of the first object detection frame and the second object detection frame can be at least one.

[0052] For example, the first image includes first object detection frames box11, box12, and box13, and the second image includes second object detection frames box21 and box22. Among them, the first object detection frame box11 and the second object detection frame box21 represent the same object. Then, one of the first object detection frame box1 and the second object detection frame box2 can be deleted, and the third object detection frame after deduplication includes box12, box13, box21, and box22.

[0053] In operation S230, object information related to the target image is determined based on the third object detection frame.

[0054] For example, after the deduplication operation, statistics can be collected on the third object detection frame corresponding to each image pair to obtain object information. The object information may include information such as the type of object and the number of each type of object.

[0055] In practical applications, when the target area is a shelf and the object is a commodity placed on the shelf, the object information can be used to determine the purchase volume, sales volume and other information of the commodity, thereby providing a reference for merchants to purchase goods.

[0056] According to the technical solution provided by the embodiments of the present disclosure, a first object detection frame and a second object detection frame are obtained from a target image through object detection. The first and second object detection frames are then removed, and object information is then counted based on the deduplicated third object detection frame. Since there is no need to stitch the target images together into a complete image and determine object information based on the complete image, the problem of poor stitching affecting object information statistics can be alleviated, thereby improving the accuracy of object information.

[0057] Figures 3A-3D Schematic diagram of candidate movement trajectories according to an embodiment of the present disclosure.

[0058] The above method for determining object information is used to process a target image, thereby obtaining object information related to the target image. It can be seen that before processing the target image, the target image can be acquired first.

[0059] In some embodiments, in order to facilitate the user to capture the target image, a front-end guiding module may be used to guide the image capture position of the image capture device. The front-end guiding module may be a software module.

[0060] First, the candidate movement trajectories of the image acquisition device can be pre-set. For a single row of shelves, the candidate movement trajectories can include one of a "left" trajectory and a "right" trajectory. For multi-row shelves, the candidate movement trajectories can include Figure 3A The "down and right" trajectory shown, Figure 3B The "up and right" trajectory shown, Figure 3C The "down and left" trajectory shown, and Figure 3D The "up and left" trajectory shown. Figures 3A-3D In FIG, the dotted arrows represent candidate movement trajectories, and images 301, 302, 303, and 304 represent four target images sequentially captured along the candidate movement trajectories.

[0061] It can be seen that when the above candidate moving trajectories are used for multiple rows of shelves, the moving trajectory of the image acquisition device is shorter, so it is easier to make the overlapping rate of two adjacent columns of images.

[0062] Secondly, the target trajectory can be determined from multiple candidate trajectories using information from the image capture. The following describes the process of determining the target trajectory, taking as an example a case where the target region includes multiple subregions arranged along a predetermined direction, where the predetermined direction can include at least one of a horizontal direction and a vertical direction.

[0063] The moving direction of the image acquisition device from a first position to a second position can be detected, where the first position is the physical position of the image acquisition device when acquiring the i-th image, and the second position is the physical position of the image acquisition device when acquiring the i+1-th image, where i is an integer greater than or equal to 1.

[0064] After the image acquisition device acquires the first image, the image acquisition device may move in any direction of up, down, left, or right to acquire the second image.

[0065] If it is detected that after capturing the first image, the image capture device moves in a horizontal direction (e.g., left or right) and captures a second image, it means that image capture is performed on a single-layer shelf. In this case, the moving direction of the first horizontal movement can be used as the target moving direction.

[0066] If it is detected that after capturing the first image, the image capture device moves in a vertical direction (e.g., upward or downward) and captures a second image, it indicates that image capture of the multi-layer shelf is being performed. If the vertical direction is upward, the first direction is recorded as "upward." If the vertical direction is downward, the first direction is recorded as "downward."

[0067] If it is detected that after capturing the i-th image, the image capture device moves horizontally for the first time and captures the i+1-th image, where i is an integer greater than or equal to 2, then the first column sub-region has been captured. If the horizontal direction is leftward, then the second direction is recorded as "leftward." If the horizontal direction is rightward, then the second direction is recorded as "rightward."

[0068] Then, the target movement trajectory of the image acquisition device can be determined by the first direction and the second direction. For example, if the first direction is "upward" and the second direction is "left", the target movement trajectory is Figure 3D The "up and left" trajectory shown.

[0069] In addition, the number of images captured before the image capture device first moves horizontally and captures images can be recorded and used as the target number. The number of images corresponding to each column of sub-areas can be limited to be the same.

[0070] Next, the front-end guidance module can guide the image acquisition device to the image acquisition position according to the target movement trajectory. In addition, when acquiring images, if the movement direction of the image acquisition device is different from the target movement direction indicated in the target movement trajectory, image acquisition can be prohibited.

[0071] The disclosed embodiment guides the user to collect images through a front-end guidance module, and is suitable for scenarios where images of multiple rows of shelves and single rows of shelves are collected. By guiding the movement trajectory of the image collection device, it is easier to obtain adjacent images with an overlapping rate that meets the requirements, thereby improving the accuracy of deduplication of the target image.

[0072] Figure 4 is a schematic flowchart of acquiring multiple images according to an embodiment of the present disclosure.

[0073] like Figure 4 As shown, in this embodiment, in order to make two adjacent images based on the acquisition order have a certain overlap rate, the method 400 for acquiring the target image may include operations S401 to S406.

[0074] In operation S401, it is determined whether there is a previous image in the set corresponding to the target image. If not, it means that the first image has not been collected, and operation S402 can be performed. If so, operation S403 can be performed.

[0075] In operation S402, in response to receiving the image acquisition instruction, the image acquisition device acquires an image at the framing position, and adds the acquired image to the set corresponding to the target image. As can be seen, the image acquired in this operation is the first image in the set.

[0076] In operation S403 , an overlap ratio between the current estimation image and a previous image for the target position is determined.

[0077] For example, the previous image may be the last image acquired in the set based on the acquisition order. A sub-region in the target region may be framed to obtain the current estimated image.

[0078] In some embodiments, features may be extracted from the current estimated image and the previous image using SIFT (Scale-Invariant Feature Transform) or other methods. Features in the current estimated image are then matched with features in the previous image using an ANN (Artificial Neural Network) or other methods. A homography matrix between the current estimated image and the previous image is then determined based on the matched feature pairs, and an overlap ratio between the current estimated image and the previous image is determined based on the homography matrix.

[0079] In some embodiments, when extracting features from the current estimated image and the previous image, features may be extracted from a partial image of the current estimated image and a partial image of the previous image, respectively, based on the relative positional relationship between the image acquisition area corresponding to the current estimated image and the image acquisition area corresponding to the previous image, thereby improving feature extraction efficiency. For example, when the image acquisition area corresponding to the current estimated image is to the left of the image acquisition area corresponding to the previous image, features may be extracted from the partial image of the current estimated image near the right boundary, and features may be extracted from the partial image of the previous image near the left boundary. The area ratio of the partial image to the complete image may be 0.3, 0.5, or the like.

[0080] In addition, in order to avoid jitter or mutation in the overlapping portion of the current estimated image and the previous image during the process of moving the image acquisition device, the overlapping portion can be stably tracked using an LK (Lucas-Kanade) optical flow tracking algorithm.

[0081] In addition, the overlapping portion may be displayed in the current estimated image through a mask, and the size and display position of the mask may change as the framing position of the current estimated image changes.

[0082] In operation S404, it is determined whether the overlap ratio is less than an overlap ratio threshold. If so, operation S405 may be performed. If not, operation S406 may be performed.

[0083] The current overlap rate may be determined every predetermined time period, and the predetermined time period may be 1 millisecond.

[0084] In operation S405 , prompt information for indicating the target moving direction is generated.

[0085] For example, if the overlap ratio is less than an overlap ratio threshold, capturing the current estimated image for the target position may be prohibited. A prompt instruction may also be generated to prompt the user to adjust the framing position of the image capture device along the target movement direction indicated by the target movement trajectory, thereby increasing the overlap ratio.

[0086] In operation S406 , an image is captured at the target location according to the received image capture instruction, and the captured image is determined as a target image and added to the set.

[0087] According to the technical solutions provided by the embodiments of the present disclosure, two adjacent images, based on the order of acquisition, have overlapping portions, thereby alleviating the problem of missing local areas within the target region during image acquisition. Furthermore, the overlap ratio between the two adjacent images can be set to be greater than or equal to an overlap ratio threshold. When deduplicating object detection frames based on the characteristics of the overlapping portions, a larger overlap ratio can improve deduplication accuracy.

[0088] According to another embodiment of the present disclosure, the method for determining object information may further include the following operations: stitching two target images obtained by capturing images of two adjacent sub-regions along a predetermined direction in a target image to obtain a stitched image, and then outputting the stitched image. In response to receiving an undo instruction, deleting the target image corresponding to the undo instruction in the target image according to the undo instruction.

[0089] like Figure 5A As shown, the predetermined direction may be a vertical direction. It can be seen that two adjacent sub-regions along the predetermined direction are arranged up and down, and the two images corresponding to the two sub-regions may be the first image 501 and the second image 502 in FIG. 5 .

[0090] During the stitching process, when two sub-regions are arranged vertically, and the first image corresponds to the upper sub-region and the second image corresponds to the lower sub-region, feature points can be extracted from the local image of the lower half of the first image and the local image of the upper half of the second image.

[0091] For example, Figure 5A As shown, feature points can be extracted from the dotted line area of ​​the first image 501 and the dotted line area of ​​the second image 502. Then, the feature points are matched to obtain feature point pairs. Figure 5A The two points connected by the same line segment in represent a matching pair of feature points. The feature point pairs can then be used to determine the homography matrix of the first image 501 and the second image 502, and the homography matrix can be used to splice the first image 501 and the second image 502.

[0092] like Figure 5B As shown, the predetermined direction may be a horizontal direction. It can be seen that two adjacent sub-regions along the predetermined direction are arranged left and right, and the two images corresponding to the two sub-regions may be Figure 5B The first image 503 and the second image 504 in FIG.

[0093] During the stitching process, when two sub-regions are arranged vertically, and the first image corresponds to the sub-region on the left and the second image corresponds to the sub-region on the right, feature points can be extracted from the right half image of the first image and the left half image of the second image.

[0094] For example, Figure 5B As shown, feature points can be distributed and extracted from the dotted-line area of ​​the first image 503 and the dotted-line area of ​​the second image 504 . Figure 5B The two points connected by the same line segment in the image 500 represent a matching feature point pair. Then, a homography matrix is ​​determined based on the matching feature point pair, and the first image 503 and the second image 504 are spliced ​​based on the homography matrix.

[0095] like Figure 5CAs shown, the predetermined direction may include a vertical direction and a horizontal direction. In this case, the multiple sub-regions are arranged in an array. At least two images obtained by capturing images of at least two sub-regions located in the same column of the array from the multiple images can be stitched together to form a stitched image. Furthermore, two adjacent columns of images can be stitched together again to form another stitched image.

[0096] For example, the first sub-region is located to the left of the second sub-region, the third sub-region is located to the left of the fourth sub-region and the two sub-regions overlap, the first sub-region is located above the third sub-region and the two sub-regions overlap, and the second sub-region is located above the fourth sub-region and the two sub-regions overlap. Images of the above four sub-regions are captured respectively to obtain images 505, 506, 507, and 508.

[0097] Images 505 and 507 can be stitched together to obtain a first column of stitched images, and images 506 and 508 can be stitched together to obtain a second column of stitched images, with reference to the stitching method for first image 501 and second image 502. The first column of stitched images and the second column of stitched images can also be stitched together to obtain a target stitched image, with reference to the stitching method for first image 503 and second image 504.

[0098] For example, at least one of the first column of stitched images, the second column of stitched images, and the target stitched image can be displayed. By displaying the stitched images, the user can intuitively understand whether the captured images are aligned. If the alignment is poor, the user can trigger an undo command to cancel the captured image and then recapture the image.

[0099] By displaying the stitched images to the user, the disclosed embodiment can facilitate the user to delete images with poor stitching effects and recapture images, thereby improving the accuracy of deduplication.

[0100] Furthermore, the disclosed embodiments can stitch only images in the same column, row, or between two adjacent columns, eliminating the need to stitch multiple columns of images into a single image. For example, the first column of images can be stitched together with the second column, or the second column of images can be stitched together with the third column, without stitching together the first column, the second column, and the third column. This simplifies the stitching process, conserves resources, and improves the real-time display of the stitched image.

[0101] According to another embodiment of the present disclosure, the above-mentioned operation of deduplicating object detection frames indicating the same object in the first object detection frame and the second object detection frame may include the following operations: determining matching feature point pairs based on the first object detection frame and the second object detection frame, each feature point pair including a first feature point in the first object detection frame and a second feature point in the second object detection frame. Then, based on the feature point pairs, determining the homography matrix of the first image and the second image. Based on the homography matrix, determining the position of the overlapping portion in the first image. Then, deleting the first object detection frame in the overlapping portion in the first object detection frame.

[0102] For example, to prevent the surrounding environment from affecting the determination of overlapping areas, this embodiment filters feature points based on the object detection frame and performs feature point matching only based on feature points within the object detection frame. A homography matrix is ​​then calculated based on multiple matching feature point pairs. This embodiment does not limit the method for calculating the homography matrix.

[0103] Next, the second image can be mapped onto the first image based on the homography matrix to obtain the overlapping portion of the second image on the first image. If the first object detection frame is completely within the overlapping portion, it can be deleted. If part of the first object detection frame is within the overlapping portion and part is outside the overlapping portion, the first object detection frame can be retained or deleted.

[0104] For example, Figure 6A As shown in the figure, in the first image, the gray area represents the overlapping portion. It can be seen that the first object detection frames 601, 602, and 605 are located outside the overlapping portion and therefore do not need to be deleted. The first object detection frames 604 and 606 are located inside the overlapping portion and therefore can be deleted. A portion of the first object detection frame 603 is located inside the overlapping portion and can be deleted or retained.

[0105] The embodiment of the present disclosure determines the position of the overlapping portion of the first image and the second image based on the matching feature point pairs in the object detection frame, and then deduplicates based on the position of the overlapping portion and the position of the object detection frame, which can improve the accuracy of deduplication.

[0106] According to another embodiment of the present disclosure, the above-mentioned operation of deduplicating object detection frames indicating the same object in the first object detection frame and the second object detection frame may include the following operations: determining a matching relationship between the first object detection frame and the second object detection frame, and then deduplicating the first object detection frame based on the matching relationship, first relative position information, and second relative position information. The first relative position information indicates the relative position between the image acquisition area corresponding to the first image and the image acquisition area corresponding to the second image; the second relative position information indicates the relative position of the first object detection frame in the first image.

[0107] For example, in response to detecting that the operation of determining the homography matrix based on the first object detection frame and the second object detection frame has failed, the operation of determining the matching relationship may be performed. For example, the homography matrix calculation failure may be determined when the number of matching feature point pairs is less than a first number threshold, and the first number threshold may be 4, 5, 10, etc.

[0108] For example, for a first object detection frame Di in a first image and a second object detection frame Dj in a second image, the similarity between the partial image corresponding to the first object detection frame Di and the partial image corresponding to the second object detection frame Dj can be calculated. If the similarity is greater than or equal to a similarity threshold, the first object detection frame Di and the second object detection frame Dj are determined to match. For example, the similarity threshold can be set to 0.6.

[0109] For another example, feature points in a first object detection frame Di can be matched with feature points in a second object detection frame Dj. If the number of matching feature point pairs meets a predetermined condition, the first object detection frame Di and the second object detection frame Dj are determined to match. The predetermined condition can include at least one of the following: the number of matching feature point pairs is greater than or equal to a second quantity threshold, and the ratio of the number of matching feature point pairs to the number of feature points in the first object detection frame Di is greater than or equal to a ratio threshold. The second quantity threshold can be 2, 3, 8, etc., and the ratio threshold can be 0.3, etc.

[0110] For example, if the first relative position information indicates that the image acquisition area corresponding to the first image is located behind the image acquisition area corresponding to the second image along a predetermined direction, the first object detection frame Di can be deleted, and the first object detection frame located behind the first object detection frame Di along the predetermined direction can also be deleted.

[0111] For example, Figure 6BAs shown, the first image includes first object bounding boxes 607, 608, 609, and 610 arranged in sequence from left to right, and the second image includes second object bounding boxes 609', 611, and 612 arranged in sequence from left to right, where the first object bounding box 609 matches the second object bounding box 609'. Since the image capture area corresponding to the first image is located to the left of the image capture area corresponding to the second image, the first object bounding boxes 609 and 610 can be deleted.

[0112] According to the matching feature point pairs in the object bounding boxes, the embodiments of the present disclosure determine the matching object bounding boxes in the first image and the second image, and then perform deduplication based on the first relative position information, the second relative position information, and the matching relationship. Therefore, even if a certain first object bounding box on the first image fails to match a second object bounding box on the second image, it can be determined whether to delete the first object bounding box according to the first relative position information and the second relative position information, thereby improving the deduplication accuracy.

[0113] According to another embodiment of the present disclosure, the operation of deduplicating the object bounding boxes indicating the same object in the first object bounding box and the second object bounding box can include the following operations: determining first object sequence information according to the relative position of the first object bounding box in the first image, determining second object sequence information according to the relative position of the second object bounding box in the second image, and then determining repeated sequence segment information in the first object sequence information and the second object sequence information. Then, according to the first relative position information and the repeated sequence segment information, the object bounding boxes indicated by the sequence segment information are deleted. The first relative position information indicates the relative position between the image capture area corresponding to the first image and the image capture area corresponding to the second image.

[0114] For example, the operation of determining the first object sequence information and the second object sequence information can be performed after the operation of determining the matching relationship between the first object bounding box and the second object bounding box is performed, and the execution result is detected. In the case where the execution result indicates that the first object bounding box and the second object bounding box cannot be matched, the operation of determining the first object sequence information and the second object sequence information is performed.

[0115] The object corresponding to the object bounding box can be represented by SKU (Stock Keeping Unit). For example, the multiple objects are multiple commodities, and the multiple commodities have different information such as color, size, etc., so the SKUs of the multiple commodities are different. The SKUs of the multiple objects are arranged in sequence according to the arrangement order of the objects, and the object sequence information can be obtained. In addition, the layer number of the object on the shelf can be determined through the shelf line, and the repeated sequence segment information can be determined according to the first object sequence information and the second object sequence information of the objects located on the same layer of the shelf.

[0116] For example, the first object sequence information is ABCDE, that is, the objects in the first image include objects A, B, C, D, and E arranged in sequence from left to right. The second object sequence information is DEBCA, that is, the objects in the second image include objects D, E, B, C, and A arranged in sequence from left to right. It can be seen that the repeated sequence segment information is BC and DE. In addition, the first relative position information indicates that the image acquisition area corresponding to the first image is located to the left of the image acquisition area corresponding to the second image. Therefore, the repeated sequence segment information near the back end of the first object sequence information can be selected, and the first object detection frame indicated by the sequence segment information can be deleted, that is, the two first object detection frames corresponding to DE are deleted.

[0117] The embodiment of the present disclosure performs deduplication based on the first relative position information and repeated sequence segment information, which can improve the accuracy of deduplication.

[0118] In some embodiments, it may also be determined whether the duplicate sequence segment information includes at least two SKUs. If not, deduplication is rejected. If so, an operation of deleting the object detection frame indicated by the sequence segment information is performed.

[0119] For example, the object sequence information of the products on the shelf is ABCCCDE, where the first object sequence information captured by the first image is ABCC, and the second object sequence information captured by the first image is CCDE. It can be seen that although there is repeated sequence segment information CC, since the repeated sequence segment information only includes one SKU (i.e., C above), the object detection box indicated by the sequence segment information CC can be retained.

[0120] Figure 7 It is a schematic structural block diagram of an apparatus for determining object information according to an embodiment of the present disclosure.

[0121] like Figure 7 As shown, the apparatus 700 for determining object information may include an image pair determination module 710 , a deduplication module 720 , and an information determination module 730 .

[0122] The image pair determination module 710 is configured to determine at least one image pair based on at least two target images, wherein the image pair includes a first image and a second image, and the first image and the second image have an overlapping portion representing the same position.

[0123] The deduplication module 720 is used to perform target detection on the first image and the second image for each image pair, respectively, to obtain a first object detection frame corresponding to the first image and a second object detection frame corresponding to the second image; and to deduplicate object detection frames indicating the same object in the first object detection frame and the second object detection frame, to obtain a third object detection frame corresponding to each image pair.

[0124] The information determination module 730 is configured to determine object information related to the target image according to the third object detection frame.

[0125] According to another embodiment of the present disclosure, the deduplication module includes: a feature point pair determination submodule, a matrix determination submodule, a position determination submodule and a first deduplication submodule. The feature point pair determination submodule is used to determine matching feature point pairs based on the first object detection frame and the second object detection frame, wherein the feature point pair includes a first feature point in the first object detection frame and a second feature point in the second object detection frame. The matrix determination submodule is used to determine the homography matrix of the first image and the second image based on the feature point pair. The position determination submodule is used to determine the position of the overlapping portion in the first image based on the homography matrix. The first deduplication submodule is used to delete the first object detection frame in the overlapping portion in the first object detection frame.

[0126] According to another embodiment of the present disclosure, a deduplication module includes: a relationship determination submodule and a second deduplication submodule. The relationship determination submodule is used to determine the matching relationship between the first object detection frame and the second object detection frame. The second deduplication submodule is used to deduplicate the first object detection frame based on the matching relationship, the first relative position information, and the second relative position information. The first relative position information indicates the relative position between the image acquisition area corresponding to the first image and the image acquisition area corresponding to the second image. The second relative position information indicates the relative position of the first object detection frame in the first image.

[0127] According to another embodiment of the present disclosure, the deduplication module includes: a first sequence determination submodule, a second sequence determination submodule and a third deduplication submodule. The first sequence determination submodule is used to determine the first object sequence information based on the relative position of the first object detection frame in the first image. The second sequence determination submodule is used to determine the second object sequence information based on the relative position of the second object detection frame in the second image. The third deduplication submodule is used to delete the first object detection frame indicated by the sequence fragment information in the first object detection frame based on the first relative position information and the repeated sequence fragment information in the first object sequence information and the second object sequence information. The first relative position information indicates the relative position between the image acquisition area corresponding to the first image and the image acquisition area corresponding to the second image.

[0128] According to another embodiment of the present disclosure, the device further includes an overlap rate determining submodule, a prompt submodule, and a capturing submodule. The overlap rate determining submodule is configured to determine an overlap rate between the current estimated image and the previous image for the target position before determining the at least one image pair. The prompt submodule is configured to generate prompt information indicating the target moving direction when the overlap rate is determined to be less than the overlap rate threshold. The capturing submodule is configured to capture an image of the target position according to the received image capturing instruction and determine the captured image as the target image when the overlap rate is determined to be greater than or equal to the overlap rate threshold.

[0129] According to another embodiment of the present disclosure, the target image is obtained by capturing at least two sub-regions in the target region, and the sub-regions are arranged along a predetermined direction. The device further includes a splicing module, an output module, and a deletion module. The splicing module is configured to splice two target images obtained by capturing two sub-regions adjacent along the predetermined direction in the target image to obtain a spliced image. The output module is configured to output the spliced image. The deletion module is configured to delete, in response to receiving a revocation instruction, a target image corresponding to the revocation instruction from the target image according to the revocation instruction.

[0130] According to another embodiment of the present disclosure, the target image is obtained by capturing at least two sub-regions in the target region, and the sub-regions are arranged as an array. The image pair determining module includes at least one of a first determining submodule and a second determining submodule. The first determining submodule is configured to determine, as an image pair, two target images obtained by capturing two sub-regions adjacent in the same column in the array in the target image. The second determining submodule is configured to determine, as an image pair, two target images obtained by capturing two sub-regions adjacent in the same row in the array in the target image.

[0131] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0132] In the technical solution of the present disclosure, the authorization or consent of the user is obtained before the user personal information is acquired or captured.

[0133] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device including at least one processor, and a memory connected with the at least one processor in communication. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method for determining object information.

[0134] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the above method for determining object information.

[0135] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, which implements the above method for determining object information when executed by a processor.

[0136] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0137] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0138] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0139] The computing unit 801 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the method for determining object information. For example, in some embodiments, the method for determining object information can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method for determining object information described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the method for determining object information by any other appropriate means (e.g., by means of firmware).

[0140] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0145] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0146] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0147] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for determining object information, comprising: Determine at least one image pair based on at least two target images, wherein the image pair includes: a first image and a second image, and the first image and the second image have an overlapping portion representing a same location; For each of the image pairs, performing object detection on the first image and the second image, respectively, to obtain a first object detection frame corresponding to the first image and a second object detection frame corresponding to the second image; and removing duplicate object detection frames indicating the same object in the first object detection frame and the second object detection frame, to obtain a third object detection frame corresponding to each of the image pairs; determining object information related to the target image according to the third object detection frame; Deduplicating object detection frames indicating the same object in the first object detection frame and the second object detection frame, including: determining a matching relationship between the first object detection frame and the second object detection frame; Deduplicating the first object detection frame according to the matching relationship, the first relative position information, and the second relative position information; Among them, the first relative position information indicates the relative position between the image acquisition area corresponding to the first image and the image acquisition area corresponding to the second image; the second relative position information indicates the relative position of the first object detection box in the first image.

2. The method according to claim 1, wherein Deduplicating object detection frames indicating the same object in the first object detection frame and the second object detection frame, further comprising: Determining a matching feature point pair based on the first object detection frame and the second object detection frame, wherein the feature point pair includes a first feature point in the first object detection frame and a second feature point in the second object detection frame; determining a homography matrix of the first image and the second image according to the feature point pairs; determining a position of the overlapping portion in the first image according to the homography matrix; and The first object detection frame in the overlapping portion is deleted from the first object detection frame.

3. The method according to claim 1, wherein Deduplicating object detection frames indicating the same object in the first object detection frame and the second object detection frame, further comprising: determining first object sequence information according to a relative position of the first object detection frame in the first image; determining second object sequence information according to a relative position of the second object detection frame in the second image; and Deleting, from the first object detection frame, the first object detection frame indicated by the sequence segment information according to the first relative position information and the repeated sequence segment information in the first object sequence information and the second object sequence information; The first relative position information indicates the relative position between the image acquisition area corresponding to the first image and the image acquisition area corresponding to the second image.

4. The method according to any one of claims 1 to 3, further comprising: Before determining at least one image pair, determining an overlap ratio between a current estimated image and a previous image for a target location; If it is determined that the overlap rate is less than the overlap rate threshold, generating prompt information for indicating the target moving direction; as well as When it is determined that the overlap rate is greater than or equal to the overlap rate threshold, image acquisition is performed on the target position according to the received image acquisition instruction, and the acquired image is determined as the target image.

5. The method according to claim 4, wherein The target image is obtained by capturing images of at least two sub-areas in the target area, and the sub-areas are arranged along a predetermined direction; the method further includes: splicing two target images obtained by capturing images of two adjacent sub-regions along the predetermined direction in the target image to obtain a spliced ​​image; outputting the stitched image; and In response to receiving a cancellation instruction, according to the cancellation instruction, the target image corresponding to the cancellation instruction is deleted from the target images.

6. The method according to claim 1, wherein The target image is obtained by capturing images of at least two sub-areas in the target area, where the sub-areas are arranged in an array; and determining at least one image pair based on the at least two target images includes at least one of the following: Determining two target images obtained by capturing images of two adjacent sub-regions located in the same column of the array as an image pair; and Two target images obtained by performing image acquisition on two adjacent sub-regions located in the same row of the array in the target image are determined as an image pair.

7. A device for determining object information, comprising: An image pair determination module is configured to determine at least one image pair based on at least two target images, wherein the image pair includes: a first image and a second image, and the first image and the second image have an overlapping portion representing a same position; a deduplication module configured to perform object detection on the first image and the second image for each of the image pairs, respectively, to obtain a first object detection frame corresponding to the first image and a second object detection frame corresponding to the second image; and to dedupe object detection frames indicating the same object in the first object detection frame and the second object detection frame, to obtain a third object detection frame corresponding to each of the image pairs; an information determining module, configured to determine object information related to the target image based on the third object detection frame; The deduplication module includes: a relationship determination submodule, configured to determine a matching relationship between the first object detection frame and the second object detection frame; a second deduplication submodule, configured to dedupe the first object detection frame according to the matching relationship, the first relative position information, and the second relative position information; Among them, the first relative position information indicates the relative position between the image acquisition area corresponding to the first image and the image acquisition area corresponding to the second image; the second relative position information indicates the relative position of the first object detection box in the first image.

8. The device according to claim 7, wherein The deduplication module further includes: a feature point pair determination submodule, configured to determine a matching feature point pair based on the first object detection frame and the second object detection frame, wherein the feature point pair includes a first feature point in the first object detection frame and a second feature point in the second object detection frame; a matrix determination submodule, configured to determine a homography matrix of the first image and the second image based on the feature point pairs; a position determination submodule, configured to determine a position of the overlapping portion in the first image according to the homography matrix; and The first deduplication submodule is configured to delete the first object detection frames in the overlapping portion from the first object detection frames.

9. The device according to claim 7, wherein The deduplication module further includes: a first sequence determination submodule, configured to determine first object sequence information according to a relative position of the first object detection frame in the first image; a second sequence determination submodule, configured to determine second object sequence information according to a relative position of the second object detection frame in the second image; and a third deduplication submodule, configured to delete, from the first object detection frame, the first object detection frame indicated by the sequence segment information, based on the first relative position information and the repeated sequence segment information in the first object sequence information and the second object sequence information; The first relative position information indicates the relative position between the image acquisition area corresponding to the first image and the image acquisition area corresponding to the second image.

10. The apparatus according to any one of claims 7 to 9, further comprising: an overlap ratio determination submodule, configured to determine an overlap ratio between a current estimated image and a previous image for a target location before determining at least one image pair; a prompt submodule, configured to generate prompt information indicating a target moving direction when it is determined that the overlap rate is less than an overlap rate threshold; as well as The acquisition submodule is configured to, when it is determined that the overlap rate is greater than or equal to the overlap rate threshold, acquire an image of the target position according to the received image acquisition instruction, and determine the acquired image as the target image.

11. The device according to claim 10, wherein The target image is obtained by collecting images of at least two sub-areas in the target area, and the sub-areas are arranged along a predetermined direction; the device further includes: a stitching module, configured to stitch two target images obtained by capturing images of two adjacent sub-regions along the predetermined direction in the target image to obtain a stitched image; an output module, configured to output the stitched image; and The deleting module is configured to, in response to receiving a revocation instruction, delete the target image corresponding to the revocation instruction in the target image according to the revocation instruction.

12. The device according to claim 7, wherein The target image is obtained by capturing images of at least two sub-areas in the target area, where the sub-areas are arranged in an array; and the image pair determination module includes at least one of the following based on the at least two target images: A first determining submodule is configured to determine, in the target image, two target images obtained by performing image acquisition on two adjacent sub-regions located in the same column of the array as an image pair; as well as The second determining submodule is configured to determine, in the target image, two target images obtained by performing image acquisition on two adjacent sub-regions located in the same row of the array as an image pair.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images