A fullness detection method and device based on target segmentation

Abstract

The application discloses a fullness detection method and device based on target segmentation, which takes a picture recording a target scene and a plurality of first containers in the target scene as a to-be-detected picture, detects and identifies the to-be-detected picture based on a target segmentation algorithm, calculates the distance from the center position of each first container to a preset corner point according to the result, assigns each first container to a corresponding corner point according to a preset condition, calculates the area of the obtained corner container and the empty space area, obtains an area ratio and a first empty space area, calculates a second empty space area according to the shooting angle of the to-be-detected picture, the area ratio and a preset empty space area, takes the average value of the fullness of all the corner containers calculated from the first empty space area and the second empty space area as the fullness of the target scene, and can quickly and accurately obtain the empty space area of the target scene and complete the fullness detection of the target scene.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and apparatus for fullness detection based on target segmentation. Background Technology

[0002] With the rapid growth of various markets, the standard for measuring market excellence is gradually shifting from single product sales volume to the end-user performance of products. For example, in the fast-moving consumer goods (FMCG) market, timely understanding and tracking of the actual inventory status of smart terminals is becoming increasingly important. Furthermore, the inventory status and sales performance of goods affect the actual utility of smart terminals, which in turn affects actual market sales. Therefore, in the past, determining whether terminal inventory was appropriate relied primarily on sales personnel conducting on-site inspections. This involved visual and manual checks to determine if containers needed restocking and to identify inventory units for best-selling products, thereby adjusting restocking strategies accordingly. This process was not only time-consuming and labor-intensive, but also inefficient, failing to track sales performance and dynamically adjust restocking strategies, thus hindering the maximization of terminal utility. Summary of the Invention

[0003] This invention provides a fullness detection method and apparatus based on target segmentation. After acquiring a photo of the target scene to be detected, the photo is segmented, and the resulting containers, empty spaces, and corresponding areas are allocated and calculated. This allows for the rapid and accurate determination of the fullness of the target scene.

[0004] To address the aforementioned technical problems, this invention provides a fullness detection method based on target segmentation, comprising:

[0005] Obtain the image to be detected; wherein, the image to be detected records the target scene and multiple first containers in the target scene;

[0006] Based on a deep learning target segmentation algorithm, each first container and its empty space in the target scene is detected, identified and segmented. The distance from the center of each first container to a preset corner point is calculated based on the results of the detection, identification and image segmentation. Each first container is assigned to the corresponding corner point according to preset conditions to obtain a corner container.

[0007] Calculate the container area and container empty area of ​​the corner container to obtain the area ratio and the first empty area;

[0008] The shooting angle of the image to be detected is obtained, and the second empty space area is calculated based on the shooting angle of the image to be detected, the area ratio, and the preset empty space area.

[0009] The container fullness is calculated from the area of ​​the first empty space and the area of ​​the second empty space. The average fullness of all the corner containers is calculated, and the average value is used as the fullness of the target scene to complete the fullness detection of the target scene.

[0010] By acquiring the image to be detected, a deep learning target segmentation algorithm is used to detect and segment the first container and the first container empty space in the target scene of the image to be detected. Then, the detection results are calculated and allocated to obtain corner containers. Next, the container area and container empty space area of ​​the corner containers are calculated to obtain the area ratio and the first empty space area. The second empty space area is calculated based on the shooting angle of the image to be detected, the area ratio, and the preset empty space area. Then, the container fullness is calculated by using the first empty space area and the second empty space area. The average of the fullness of all containers is taken as the fullness of the target scene. This method can quickly and accurately obtain the empty space area of ​​the target scene and complete the fullness detection of the target scene.

[0011] As a preferred embodiment, the step of assigning each of the first containers to the corresponding corner points according to preset conditions to obtain corner point containers includes:

[0012] Based on the empty space of the first container, the first container is filtered to obtain a container containing the empty space of the first container, which is set as the second container, and the center of the second container is obtained;

[0013] Based on the four preset points (top left, top right, bottom left, and bottom right), compare the number of centers of the second container with the total number of preset corner points. When the number of centers of the second container is greater than the total number of preset corner points, calculate the mask area of ​​all the second containers and sort them from largest to smallest. According to the order of the mask areas of the second containers, assign the centers of the first four second containers to the preset corner point with the smallest distance to them, thus obtaining the corner container.

[0014] By calculating and sorting the minimum distance between the preset corner point and the center of the first container, and assigning the corresponding preset corner point to the center of the second container, the corner point container is obtained. This can be used to determine that the obtained corner point container is an accurate container space, avoiding erroneous detection results caused by not detecting any containers.

[0015] As a preferred embodiment, the step of filtering the first container based on the first container empty space to obtain a container containing the first container empty space specifically involves:

[0016] The first container and the first container empty space are obtained based on the image segmentation result. The overlap between the first container mask and the first container empty space mask is obtained by bit operation. When the overlap is greater than a preset value, the first container is considered to contain the first container empty space.

[0017] The filtering method described above allows for the setting of key locations based on terminal requirements using preset comparison values, resulting in a final outcome that better meets actual needs.

[0018] As a preferred embodiment, the calculation of the container area and container empty space area of ​​the corner container to obtain the area ratio and the first empty space area is specifically as follows:

[0019] Calculate the area of ​​each corner container and sort them by size. Set the ratio of the largest and smallest areas of the calculated corner containers as the area ratio. Calculate the empty space area of ​​each corner container, where the first empty space area includes the empty space areas of all corner containers.

[0020] By calculating the area ratio and the area of ​​the first empty space, it can be used to determine the shooting angle and calculate the fullness of the container, making the final fullness more accurate.

[0021] As a preferred embodiment, obtaining the shooting angle of the image to be detected specifically involves:

[0022] The area ratio is compared with a preset first threshold. If the area ratio is greater than the first threshold, the shooting angle is set to a first angle; if the area ratio is less than the first threshold, the shooting angle is set to a second angle.

[0023] By setting different shooting angles for the image to be detected, it is possible to adapt the shooting scenes obtained from different angles.

[0024] As a preferred embodiment, the calculation of the second empty space area based on the shooting angle, the area ratio, and the preset empty space area specifically involves:

[0025] The preset empty space area includes a second threshold, a third threshold, and a fourth threshold, wherein the second threshold, the third threshold, and the fourth threshold are set by preset debugging experience values;

[0026] The second vacancy area includes the first estimated vacancy area, the second estimated vacancy area, and the third estimated vacancy area;

[0027] When the shooting angle is the first angle, the estimated total empty space area of ​​each corner container is calculated according to the second threshold to obtain the first estimated empty space area;

[0028] When the shooting angle is the second angle, the estimated total empty space area of ​​the upper left corner container and the upper right corner container is calculated according to the third threshold to obtain the second estimated empty space area. Then, the estimated total empty space area of ​​the lower left corner container and the lower right corner container is calculated according to the fourth threshold to obtain the third estimated empty space area.

[0029] By processing scenes from different shooting angles accordingly, we can avoid the error caused by using the same processing method on scenes with different shooting angles.

[0030] As a preferred embodiment, the calculation of container fullness from the area of ​​the first empty space and the area of ​​the second empty space specifically involves:

[0031] The fullness of each corner container is calculated using the following formula, based on the area of ​​its first empty space and the area of ​​its corresponding second empty space:

[0032] R ci_full_ratio =1-max(0,min(A) ci_e / A ci_all ,1))

[0033] Among them, R ci_full_ratio A represents the fullness of the corner container. ci_e A is the area of ​​the first empty space in each of the corner containers. ci_all The area of ​​the second empty space corresponding to each corner container;

[0034] The first area is the area of ​​empty space in the container captured by the camera, and the second empty space is the area of ​​empty space in the container obtained by preset debugging experience value. The container fullness calculated by the first empty space area and the second empty space area can not only represent the inventory of the terminal, but also be used as an estimate of the total inventory of goods in the terminal.

[0035] As a preferred embodiment, after completing the fullness detection of the target scene, the method further includes:

[0036] The inventory of goods detected by the terminal or server is used to estimate the total inventory of goods. When the inventory of goods on the terminal or server is lower than a preset threshold, the fullness is transmitted to the business system so that the user can respond to the fullness transmitted by the business system and the obtained fullness is returned to the caller in the form of service parameters.

[0037] By using the saturation level of the target scenario as an estimate of the total inventory of goods, a corresponding response can be made based on the inventory level, which can be used to implement a better product deployment strategy.

[0038] The present invention also provides a fullness detection device based on target segmentation, including an image acquisition module, a container processing module, an area acquisition module, and a fullness acquisition module;

[0039] The image acquisition module is used to acquire an image to be detected; wherein, the image to be detected records a target scene and multiple first containers in the target scene;

[0040] The container processing module is used to detect, identify and segment the first container and the empty space of the first container in the target scene based on the deep learning target segmentation algorithm. It calculates the distance from the center position of each first container to a preset corner point according to the detection and segmentation results, and assigns each container to the corresponding corner point according to preset conditions to obtain corner container.

[0041] The area acquisition module is used to calculate the container area and container empty area of ​​the corner container to obtain the area ratio and the first empty area; to obtain the shooting angle of the image to be detected, and to calculate the second empty area based on the shooting angle of the image to be detected, the area ratio and the preset empty area.

[0042] The fullness acquisition module is used to calculate the container fullness from the first empty space area and the second empty space area, calculate the average fullness of all the corner containers, and use the average value as the fullness of the target scene to complete the fullness detection of the target scene.

[0043] The image to be detected is acquired using the target segmentation-based fullness detection device. A deep learning target segmentation algorithm is used to detect and segment the first container and its empty spaces within the target scene in the image. The detection results are then calculated and allocated to obtain corner containers. Next, the area of ​​each corner container and the area of ​​its empty spaces are calculated to obtain the area ratio and the area of ​​the first empty space. Based on the shooting angle of the image, the area ratio, and the preset empty space area, the area of ​​the second empty space is calculated. Finally, the fullness of the container is calculated using the area of ​​the first empty space and the area of ​​the second empty space. The average fullness of all containers is taken as the fullness of the target scene. This method allows for rapid and accurate detection of the fullness, yielding the accurate empty space area of ​​the target scene.

[0044] As a preferred embodiment, the container processing module is used to assign each of the first containers to the corresponding corner points according to preset conditions to obtain corner point containers, including:

[0045] The container processing module filters the first container based on the first container empty space, obtains a container containing the first container empty space, sets it as the second container, and obtains the center of the second container;

[0046] The container processing module also compares the number of centers of the second container with the total number of preset corner points based on four preset points: the upper left corner, the upper right corner, the lower left corner, and the lower right corner. When the number of centers of the second container is greater than the total number of preset corner points, the module calculates the mask area of ​​all the second containers and sorts them from largest to smallest. Based on the order of the mask areas of the second containers, the modules assign the centers of the first four second containers to the preset corner point with the smallest distance to them, thus obtaining the corner container.

[0047] The container processing module calculates and sorts the minimum distance between the preset corner point and the center of the first container, and assigns the corresponding preset corner point to the center of the second container to obtain the corner point container. This can be used to determine that the obtained corner point container is an accurate container space, avoiding erroneous detection results caused by not detecting any containers.

[0048] As a preferred embodiment, the container processing module filters the first containers based on the first container empty space to obtain containers that contain the first container empty space, specifically as follows:

[0049] The image acquisition module obtains the mask of the first container and the mask of the first container empty space based on the image segmentation result, and obtains the overlap degree of the mask of the first container and the mask of the first container empty space through bit operation. When the overlap degree is greater than a preset value, it is considered that the first container contains the first container empty space.

[0050] Through the aforementioned filtering steps, the locations that require special attention can be set according to the terminal's requirements by using preset comparison values, so that the final result is more in line with actual needs.

[0051] As a preferred embodiment, the area acquisition module is used to calculate the container area and the empty space area of ​​the corner container to obtain the area ratio and the first empty space area, specifically:

[0052] The area acquisition module calculates the area of ​​each corner container, sorts them by size, and sets the ratio of the largest and smallest areas of the calculated corner containers as the area ratio; it also calculates the empty space area of ​​each corner container, where the first empty space area includes the empty space areas of all corner containers.

[0053] By calculating the area ratio and the area of ​​the first empty space, it can be used to determine the shooting angle and calculate the fullness of the container, making the final fullness more accurate.

[0054] As a preferred embodiment, the area acquisition module acquires the shooting angle of the image to be detected, specifically as follows:

[0055] The area acquisition module compares the area ratio with a preset first threshold. If the area ratio is greater than the first threshold, the shooting angle is set to a first angle; if the area ratio is less than the first threshold, the shooting angle is set to a second angle.

[0056] By setting different shooting angles for the image to be detected, it is possible to adapt the shooting scenes obtained from different angles.

[0057] As a preferred embodiment, the area acquisition module calculates the second empty space area based on the shooting angle, the area ratio, and the preset empty space area, specifically as follows:

[0058] In the area acquisition module, the preset empty space area includes a second threshold, a third threshold, and a fourth threshold, wherein the second threshold, the third threshold, and the fourth threshold are set by preset debugging experience values;

[0059] The second vacancy area includes the first estimated vacancy area, the second estimated vacancy area, and the third estimated vacancy area;

[0060] When the shooting angle is the first angle, the estimated total empty space area of ​​each corner container is calculated according to the second threshold to obtain the first estimated empty space area.

[0061] When the shooting angle is the second angle, the estimated total empty space area of ​​the upper left corner container and the upper right corner container is calculated according to the third threshold to obtain the second estimated empty space area. Then, the estimated total empty space area of ​​the lower left corner container and the lower right corner container is calculated according to the fourth threshold to obtain the third estimated empty space area.

[0062] By using the area acquisition module to process scenes from different shooting angles, errors caused by using the same processing method to process scenes from different shooting angles can be avoided.

[0063] As a preferred embodiment, the fullness acquisition module calculates the container fullness from the area of ​​the first empty space and the area of ​​the second empty space, specifically as follows:

[0064] The fullness acquisition module calculates the fullness of each corner container using the following formula, based on the area of ​​the first empty space and the area of ​​the corresponding second empty space:

[0065] R ci_full_ratio =1-max(0,min(A) ci_e / A ci_all ,1))

[0066] Among them, R ci_full_ratio A represents the fullness of the corner container. ci_e A is the area of ​​the first empty space in each of the corner containers. ci_all The area of ​​the second empty space corresponding to each corner container;

[0067] The first area is the area of ​​empty space in the container captured by the camera, and the second empty space is the area of ​​empty space in the container obtained by preset debugging experience value. The container fullness calculated by the first empty space area and the second empty space area can not only represent the inventory of the terminal, but also be used as an estimate of the total inventory of goods in the terminal.

[0068] As a preferred embodiment, the fullness detection device further includes an information feedback module, specifically defined as follows;

[0069] The information feedback module uses the product inventory detected by the terminal or server as an estimate of the total product inventory. When the product inventory on the terminal or server is lower than a preset threshold, the fullness is transmitted to the business system so that the user can respond to the fullness transmitted by the business system and the obtained fullness is returned to the caller in the form of service parameters.

[0070] By responding to the saturation of the target scenario based on the estimated inventory level, a better product deployment strategy can be implemented. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating an embodiment of the fullness detection method based on target segmentation in this invention.

[0072] Figure 2 : This is a schematic diagram of an embodiment of the fullness detection device based on target segmentation in this invention. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] Example 1

[0075] Please refer to Figure 1 This is a flowchart illustrating an embodiment of the fullness detection method based on target segmentation in this invention, including steps 101 to 104, each step as follows:

[0076] Step 101: Obtain the image to be detected; wherein the image to be detected records the target scene and multiple first containers in the target scene;

[0077] In this embodiment, the target scene may include, but is not limited to, a freezer scene, and the plurality of first containers may include, but is not limited to, a plurality of containers in a freezer; after the freezer image to be detected is acquired, it can be prepared for subsequent detection steps.

[0078] Step 102: Based on the deep learning target segmentation algorithm, detect, identify and segment each first container and the empty space of the first container in the target scene. Calculate the distance from the center position of each first container to a preset corner point according to the detection and segmentation results. Assign each first container to the corresponding corner point according to preset conditions to obtain a corner container.

[0079] In this embodiment, the step of assigning each of the first containers to the corresponding corner points according to preset conditions to obtain corner point containers includes:

[0080] Based on the empty space in the first container, the first container is filtered to obtain a container containing the empty space in the first container, which is set as the second container, and the center of the second container is obtained;

[0081] Based on four preset points—top left, top right, bottom left, and bottom right—the number of centers of the second container is compared with the total number of preset corner points. When the number of centers of the second container is greater than the total number of preset corner points, the mask area of ​​all the second containers is calculated and sorted from largest to smallest. Specifically, the countNonZero function of OpenCV can be used for calculation. According to the order of the mask areas of the second containers, the centers of the first four second containers are assigned to the preset corner points with the smallest distance to them, thus obtaining the corner container.

[0082] In this embodiment, the step of filtering the first container based on the first container empty space to obtain a container containing the first container empty space specifically involves:

[0083] The first container and the first container empty space are obtained based on the image segmentation result. The overlap between the first container mask and the first container empty space mask is obtained by bit operation. When the overlap is greater than a preset value, the first container is considered to contain the first container empty space.

[0084] By calculating and sorting the minimum distance between the preset corner points and the center of the first container, and assigning the corresponding preset corner points to the center of the second container, the corner point container is obtained. This can be used to determine that the obtained corner point container is an accurate container space, avoiding erroneous detection results caused by not detecting containers in the freezer. Through the filtering method, according to the requirements of the terminal, the key locations that need to be focused on can be set accordingly by using preset comparison values, such as the most conspicuous locations in the freezer used to place high-profit products, so that the final result is more in line with actual needs.

[0085] Step 103: Calculate the container area and container empty area of ​​the corner container to obtain the area ratio and the first empty area;

[0086] The shooting angle of the image to be detected is obtained, and the second empty space area is calculated based on the shooting angle of the image to be detected, the area ratio, and the preset empty space area.

[0087] In this embodiment, the calculation of the container area and container empty space area of ​​the corner container to obtain the area ratio and the first empty space area is specifically as follows:

[0088] Calculate the area of ​​each corner container and sort them by size. Set the ratio of the largest and smallest areas of the calculated corner containers as the area ratio. Calculate the empty space area of ​​each corner container, where the first empty space area includes the empty space areas of all corner containers.

[0089] In this embodiment, obtaining the shooting angle of the image to be detected specifically involves:

[0090] The area ratio is compared with a preset first threshold. If the area ratio is greater than the first threshold, the shooting angle is set to a first angle, which can be the angle of shooting the freezer container from the front. If the area ratio is less than the first threshold, the shooting angle is set to a second angle, which can be the angle of shooting the freezer container from the side.

[0091] In this embodiment, the calculation of the second empty space area based on the shooting angle, the area ratio, and the preset empty space area specifically involves:

[0092] The preset empty space area includes a second threshold, a third threshold, and a fourth threshold, wherein the second threshold, the third threshold, and the fourth threshold are set by preset debugging experience values;

[0093] The second vacancy area includes the first estimated vacancy area, the second estimated vacancy area, and the third estimated vacancy area;

[0094] When the shooting angle is the first angle, the estimated total empty space area of ​​each corner container is calculated according to the second threshold to obtain the first estimated empty space area;

[0095] When the shooting angle is the second angle, the estimated total empty space area of ​​the upper left corner container and the upper right corner container is calculated according to the third threshold to obtain the second estimated empty space area. Then, the estimated total empty space area of ​​the lower left corner container and the lower right corner container is calculated according to the fourth threshold to obtain the third estimated empty space area.

[0096] By calculating the area ratio and the area of ​​the first empty space, the subsequent shooting angle can be used to determine the fullness of the container, making the final fullness more accurate. By setting different shooting angles for the image to be detected, it is possible to adapt the shooting scenes obtained from different angles. For example, the container obtained by shooting the same freezer scene from the front may have a very different shape than the container obtained by shooting from the side. By processing the two cases separately, the error caused by using the same processing method to process scenes with different shooting angles can be avoided.

[0097] Step 104: Calculate the container fullness from the area of ​​the first empty space and the area of ​​the second empty space, calculate the average fullness of all the corner containers, and use the average value as the fullness of the target scene to complete the fullness detection of the target scene.

[0098] In this embodiment, the calculation of container fullness from the area of ​​the first empty space and the area of ​​the second empty space specifically involves:

[0099] The fullness of each corner container is calculated using the following formula, based on the area of ​​its first empty space and the area of ​​its corresponding second empty space:

[0100] R ci_full_ratio =1-max(0,min(A) ci_e / A ci_all ,1))

[0101] Among them, R ci_full_ratio A represents the fullness of the corner container. ci_e A is the area of ​​the first empty space in each of the corner containers. ci_all The area of ​​the second empty space corresponding to each corner container;

[0102] The calculated saturation level of the target scenario can be converted into a percentage to represent the inventory level of the scenario. For example, a saturation level of 0.35 can be converted into 35%, which means that only 35% of the inventory remains in the scenario.

[0103] In this embodiment, after completing the fullness detection of the target scene, the method further includes:

[0104] The inventory of goods detected by the terminal or server is used to estimate the total inventory of goods. When the inventory of goods on the terminal or server is lower than a preset threshold, the fullness is transmitted to the business system so that the user can respond to the fullness transmitted by the business system and the obtained fullness is returned to the caller in the form of service parameters.

[0105] The first area is the area of ​​empty container space captured in the photograph, and the second empty area is the area of ​​empty container space obtained from preset debugging experience values. The container fullness calculated by the first empty area and the second empty area can not only represent the terminal's inventory, but also be used as an estimate of the terminal's total inventory. Furthermore, by using the fullness of the target scene as an estimate of the total inventory, the actual inventory status of the terminal's goods can be provided in a timely and effective manner, and corresponding responses can be made based on the inventory, thereby quickly and accurately obtaining the actual sales status of the terminal's goods and implementing a better product deployment strategy.

[0106] Accordingly, please refer to Figure 2 This is a schematic diagram of an embodiment of a fullness detection device based on target segmentation provided in this invention, including an image acquisition module 201, a container processing module 202, an area acquisition module 203, and a fullness acquisition module 204.

[0107] The image acquisition module 201 is used to acquire an image to be detected; wherein, the image to be detected records a target scene and multiple first containers in the target scene;

[0108] In this embodiment, the target scene may include, but is not limited to, a freezer scene, and the plurality of first containers may include, but is not limited to, a plurality of containers in a freezer; after the freezer image to be detected is acquired, it can be prepared for subsequent detection steps.

[0109] The container processing module 202 is used to detect, identify and segment the first container and the first container empty space in the target scene based on the deep learning target segmentation algorithm, calculate the distance from the center position of each first container to a preset corner point according to the detection and segmentation results, and assign each container to the corresponding corner point according to preset conditions to obtain corner point containers.

[0110] In this embodiment, the container processing module 202 assigns each of the first containers to the corresponding corner points according to preset conditions to obtain corner point containers, including:

[0111] The container processing module 202 filters the first container according to the first container empty space, obtains a container containing the first container empty space, sets it as the second container, and obtains the center of the second container;

[0112] The container processing module 202 compares the number of centers of the second container with the total number of preset corner points based on four preset points: the top left corner, the top right corner, the bottom left corner, and the bottom right corner. When the number of centers of the second container is greater than the total number of preset corner points, the module calculates the mask area of ​​all the second containers and sorts them from largest to smallest. Specifically, the module can use the OpenCV countNonZero function to calculate this. Based on the order of the mask areas of the second containers, the module assigns the centers of the first four second containers to the preset corner point with the smallest distance to them, thus obtaining the corner container.

[0113] In this embodiment, the container processing module 202 further filters the first container based on the first container empty space to obtain containers containing the first container empty space, specifically:

[0114] The container processing module 202 obtains the mask of the first container and the mask of the first container empty space according to the image segmentation result, and obtains the overlap degree of the mask of the first container and the mask of the first container empty space through bit operation. When the overlap degree is greater than a preset value, it is considered that the first container contains the first container empty space.

[0115] The container processing module 202 calculates and sorts the minimum distance between the preset corner points and the center of the first container, and assigns the corresponding preset corner points to the center of the second container to obtain the corner point container. This can be used to determine that the obtained corner point container is an accurate container space, avoiding erroneous detection results caused by not detecting containers in the freezer. Through the filtering method, according to the requirements of the terminal, the location points that need to be focused on can be set accordingly by using preset comparison values, such as the most conspicuous location points in the freezer used to place high-profit products, so that the final result is more in line with actual needs.

[0116] The area acquisition module 203 is used to calculate the container area and container empty area of ​​the corner container to obtain the area ratio and the first empty area; to acquire the shooting angle of the image to be detected, and to calculate the second empty area based on the shooting angle of the image to be detected, the area ratio and the preset empty area.

[0117] In this embodiment, the area acquisition module 203 is used to calculate the container area and container empty space area of ​​the corner container to obtain the area ratio and the first empty space area, specifically:

[0118] The area acquisition module 203 calculates the area of ​​each corner container, sorts them by size, and sets the ratio of the largest and smallest areas of the calculated corner containers as the area ratio; it also calculates the empty space area of ​​each corner container, where the first empty space area includes the empty space areas of all corner containers.

[0119] In this embodiment, obtaining the shooting angle of the image to be detected specifically involves:

[0120] The area acquisition module 203 compares the area ratio with a preset first threshold. If the area ratio is greater than the first threshold, the shooting angle is set to a first angle, which can be the angle of shooting the freezer container from the front. If the area ratio is less than the first threshold, the shooting angle is set to a second angle, which can be the angle of shooting the freezer container from the side.

[0121] In this embodiment, the area acquisition module 203 calculates the second empty space area based on the shooting angle, the area ratio, and the preset empty space area, specifically as follows:

[0122] In the area acquisition module 203, the preset empty space area includes a second threshold, a third threshold, and a fourth threshold, wherein the second threshold, the third threshold, and the fourth threshold are set by preset debugging experience values;

[0123] The second vacancy area includes the first estimated vacancy area, the second estimated vacancy area, and the third estimated vacancy area;

[0124] When the shooting angle is the first angle, the estimated total empty space area of ​​each corner container is calculated according to the second threshold to obtain the first estimated empty space area.

[0125] When the shooting angle is the second angle, the estimated total empty space area of ​​the upper left corner container and the upper right corner container is calculated according to the third threshold to obtain the second estimated empty space area. Then, the estimated total empty space area of ​​the lower left corner container and the lower right corner container is calculated according to the fourth threshold to obtain the third estimated empty space area.

[0126] The area acquisition module 203 calculates the area ratio and the area of ​​the first empty space, which can be used for subsequent shooting angle judgment and container fullness calculation, making the final fullness more accurate. By setting different shooting angles for the image to be detected, it can be used to adapt the shooting scene with different angles. For example, the container obtained by shooting the same freezer scene from the front may have a very different shape than the container obtained by shooting from the side. By processing the two cases separately, the error caused by using the same processing method to process scenes with different shooting angles can be avoided.

[0127] The fullness acquisition module 204 is used to calculate the fullness of the container from the area of ​​the first empty space and the area of ​​the second empty space, calculate the average fullness of all the corner containers, and use the average value as the fullness of the target scene to complete the fullness detection of the target scene.

[0128] In this embodiment, the fullness acquisition module 204 calculates the container fullness from the area of ​​the first empty space and the area of ​​the second empty space, specifically as follows:

[0129] The fullness acquisition module 204 calculates the fullness of each corner container based on the area of ​​the first empty space and the area of ​​the second empty space corresponding to that container, using the following formula:

[0130] R ci_full_ratio =1-max(0,min(A) ci_e / A ci_all ,1))

[0131] Among them, R ci_full_ratio A represents the fullness of the corner container. ci_e A is the area of ​​the first empty space in each of the corner containers. ci_all The area of ​​the second empty space corresponding to each corner container;

[0132] The fullness of the target scene calculated by the fullness acquisition module 204 can be converted into a percentage to represent the inventory of the scene. For example, if the calculated fullness value is 0.35, it can be converted into 35%, which means that there is only 35% of the inventory left in the scene.

[0133] In this embodiment, the fullness detection device further includes an information feedback module, specifically defined as follows;

[0134] The information feedback module uses the product inventory detected by the terminal or server as an estimate of the total product inventory. When the product inventory on the terminal or server is lower than a preset threshold, the fullness is transmitted to the business system so that the user can respond to the fullness transmitted by the business system and the obtained fullness is returned to the caller in the form of service parameters.

[0135] The first area is the area of ​​empty space in the captured container, and the second empty space area is the area of ​​empty space in the container obtained from preset debugging experience values. The fullness acquisition module calculates the container fullness using the first empty space area and the second empty space area, which allows the container fullness to not only represent the terminal's inventory quantity but also to be used as an estimate of the terminal's total inventory quantity. Furthermore, by using the fullness of the target scene as an estimate of the total inventory quantity of goods, the actual inventory status of the terminal's goods can be provided in a timely and effective manner, and corresponding responses can be made based on the inventory quantity, thereby quickly and accurately obtaining the actual sales status of the terminal's goods and implementing a better product deployment strategy.

[0136] In summary, this invention provides a fullness detection method and apparatus based on target segmentation. The method uses an image recording a target scene and multiple first containers within that scene as the image to be detected. Based on a target segmentation algorithm, the image to be detected is detected, identified, and segmented. The distance from the center position of each first container to a preset corner point is calculated based on the results. Each first container is assigned to its corresponding corner point according to preset conditions to obtain corner containers. The container area and empty space area of ​​each corner container are calculated to obtain an area ratio and a first empty space area. The shooting angle of the image to be detected is obtained, and a second fullness detection method is calculated based on the shooting angle, the area ratio, and the preset empty space area. Empty space area; the container fullness is calculated from the first empty space area and the second empty space area. The average fullness of all corner containers is calculated and used as the fullness of the target scene. This allows for quick and accurate fullness detection of the target scene, obtaining the accurate empty space area of ​​the target scene. This not only represents the terminal's inventory quantity but can also be used to estimate the total inventory quantity of goods at the terminal. Furthermore, by using the fullness of the target scene as an estimate of the total inventory quantity of goods, the actual inventory status of goods at the terminal can be provided in a timely and effective manner. Based on the inventory quantity, corresponding responses can be made, thereby quickly and accurately obtaining the actual sales status of goods at the terminal and implementing a better product deployment strategy.

[0137] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for fullness detection based on target segmentation, characterized in that, include: Obtain the image to be detected; wherein, the image to be detected records the target scene and multiple first containers in the target scene; Based on a deep learning target segmentation algorithm, each first container and the empty space of the first container in the target scene are detected, identified and segmented. The distance from the center position of each first container to a preset corner point is calculated according to the results of the detection, identification and image segmentation. Each first container is assigned to the corresponding corner point according to preset conditions to obtain a corner container. Calculate the container area and empty space area of ​​the corner container to obtain the area ratio and the first empty space area; The shooting angle of the image to be detected is obtained, and the second empty space area is calculated based on the shooting angle of the image to be detected, the area ratio, and the preset empty space area. The container fullness is calculated from the area of ​​the first empty space and the area of ​​the second empty space. The average fullness of all the corner containers is calculated, and the average value is used as the fullness of the target scene to complete the fullness detection of the target scene.

2. The fullness detection method based on target segmentation as described in claim 1, characterized in that, The step of assigning each of the first containers to the corresponding corner points according to preset conditions to obtain corner point containers includes: Based on the empty space in the first container, the first container is filtered to obtain a container containing the empty space in the first container, which is set as the second container, and the center of the second container is obtained; Based on four preset points—upper left, upper right, lower left, and lower right—the number of centers of the second container is compared with the total number of preset corner points. When the number of centers of the second container is greater than the total number of preset corner points, the mask area of ​​all the second containers is calculated and sorted from largest to smallest. According to the order of the mask areas of the second containers, the centers of the first four second containers are assigned to the preset corner point with the smallest distance to them, thus obtaining the corner container.

3. The fullness detection method based on target segmentation as described in claim 2, characterized in that, The step of filtering the first container based on the first container empty space to obtain a container containing the first container empty space specifically involves: The first container and the mask of the first container empty space are obtained based on the image segmentation result. The overlap between the mask of the first container and the mask of the first container empty space is obtained by bit operation. When the overlap is greater than a preset value, the first container is considered to contain the first container empty space.

4. The fullness detection method based on target segmentation as described in claim 1, characterized in that, The calculation of the container area and container empty space area of ​​the corner container to obtain the area ratio and the first empty space area is specifically as follows: Calculate the area of ​​each corner container and sort them by size. Set the ratio of the largest and smallest areas of the calculated corner containers as the area ratio. Calculate the empty space area of ​​each corner container, where the first empty space area includes the empty space areas of all corner containers.

5. The fullness detection method based on target segmentation as described in claim 1, characterized in that, The specific steps for obtaining the shooting angle of the image to be detected are as follows: The area ratio is compared with a preset first threshold. If the area ratio is greater than the first threshold, the shooting angle is set to a first angle; if the area ratio is less than the first threshold, the shooting angle is set to a second angle.

6. The fullness detection method based on target segmentation as described in claim 1, characterized in that, The calculation of the second empty space area based on the shooting angle, the area ratio, and the preset empty space area is specifically as follows: The preset empty space area includes a second threshold, a third threshold, and a fourth threshold, wherein the second threshold, the third threshold, and the fourth threshold are set by preset debugging experience values; The second vacancy area includes the first estimated vacancy area, the second estimated vacancy area, and the third estimated vacancy area; When the shooting angle is the first angle, the estimated total empty space area of ​​each corner container is calculated according to the second threshold to obtain the first estimated empty space area. When the shooting angle is the second angle, the estimated total empty space area of ​​the upper left corner container and the upper right corner container is calculated according to the third threshold to obtain the second estimated empty space area. Then, the estimated total empty space area of ​​the lower left corner container and the lower right corner container is calculated according to the fourth threshold to obtain the third estimated empty space area.

7. The fullness detection method based on target segmentation as described in claim 1, characterized in that, The container fullness is calculated from the area of ​​the first empty space and the area of ​​the second empty space, specifically as follows: The fullness of each corner container is calculated using the following formula, based on the area of ​​its first empty space and the area of ​​its corresponding second empty space: R ci_full_ratio =1_max(0,min(A ci_e / A ci_all ,1)) Among them, R ci_full_ratio A represents the fullness of the corner container. ci_e A is the area of ​​the first empty space in each of the corner containers. ci_all The area of ​​the second empty space corresponding to each corner container.

8. The fullness detection method based on target segmentation as described in claim 1, characterized in that, After completing the fullness detection of the target scene, the following steps are also included: The inventory of goods detected by the terminal or server is used to estimate the total inventory of goods. When the inventory of goods on the terminal or server is lower than a preset threshold, the fullness is transmitted to the business system so that the user can respond to the fullness transmitted by the business system and the obtained fullness is returned to the caller in the form of service parameters.

9. A fullness detection device based on target segmentation, characterized in that, include: Image acquisition module, container processing module, area acquisition module, and fullness acquisition module; The image acquisition module is used to acquire an image to be detected; wherein, the image to be detected records a target scene and multiple first containers in the target scene; The container processing module is used to detect, identify and segment the first container and the empty space of the first container in the target scene based on the deep learning target segmentation algorithm. It calculates the distance from the center position of each first container to a preset corner point according to the detection and segmentation results, and assigns each container to the corresponding corner point according to preset conditions to obtain corner container. The area acquisition module is used to calculate the container area and container empty area of ​​the corner container to obtain the area ratio and the first empty area; to obtain the shooting angle of the image to be detected, and to calculate the second empty area based on the shooting angle of the image to be detected, the area ratio and the preset empty area. The fullness acquisition module is used to calculate the fullness of the container from the area of ​​the first empty space and the area of ​​the second empty space, calculate the average fullness of all the corner containers, and use the average value as the fullness of the target scene to complete the fullness detection of the target scene.

10. The fullness detection device based on target segmentation as described in claim 9, characterized in that, The container processing module is used to assign each of the first containers to the corresponding corner points according to preset conditions, to obtain corner point containers, including: The container processing module filters the first container based on the first container empty space, obtains a container containing the first container empty space, sets it as the second container, and obtains the center of the second container; The container processing module further compares the number of centers of the second container with the total number of preset corner points based on four preset points: the upper left corner, the upper right corner, the lower left corner, and the lower right corner. When the number of centers of the second container is greater than the total number of preset corner points, the module calculates the mask area of ​​all the second containers and sorts them from largest to smallest. Based on the order of the mask areas of the second containers, the modules assign the centers of the first four second containers to the preset corner point with the smallest distance from them, thus obtaining the corner container.

11. The fullness detection device based on target segmentation as described in claim 10, characterized in that, The container processing module filters the first containers based on the first container empty space to obtain containers that contain the first container empty space, specifically: The container processing module obtains the mask of the first container and the mask of the first container empty space based on the image segmentation result, and obtains the overlap degree of the mask of the first container and the mask of the first container empty space through bit operation. When the overlap degree is greater than a preset value, it is considered that the first container contains the first container empty space.

12. The fullness detection device based on target segmentation as described in claim 9, characterized in that, The area acquisition module is used to calculate the container area and empty space area of ​​the corner container to obtain the area ratio and the first empty space area, specifically: The area acquisition module calculates the area of ​​each corner container and sorts them by size. The ratio of the largest area to the smallest area of ​​the calculated corner containers is set as the area ratio. The module also calculates the empty space area of ​​each corner container, where the first empty space area includes the empty space areas of all corner containers.

13. The fullness detection device based on target segmentation as described in claim 9, characterized in that, The area acquisition module acquires the shooting angle of the image to be detected, specifically: The area acquisition module compares the area ratio with a preset first threshold. If the area ratio is greater than the first threshold, the shooting angle is set to a first angle; if the area ratio is less than the first threshold, the shooting angle is set to a second angle.

14. The fullness detection device based on target segmentation as described in claim 9, characterized in that, The area acquisition module calculates the second empty space area based on the shooting angle, the area ratio, and the preset empty space area, specifically as follows: In the area acquisition module, the preset empty space area includes a second threshold, a third threshold, and a fourth threshold, wherein the second threshold, the third threshold, and the fourth threshold are set by preset debugging experience values; The second vacancy area includes the first estimated vacancy area, the second estimated vacancy area, and the third estimated vacancy area; When the shooting angle is the first angle, the estimated total empty space area of ​​each corner container is calculated according to the second threshold to obtain the first estimated empty space area. When the shooting angle is the second angle, the estimated total empty space area of ​​the upper left corner container and the upper right corner container is calculated according to the third threshold to obtain the second estimated empty space area. Then, the estimated total empty space area of ​​the lower left corner container and the lower right corner container is calculated according to the fourth threshold to obtain the third estimated empty space area.

15. The fullness detection device based on target segmentation as described in claim 9, characterized in that, The fullness acquisition module calculates the container fullness from the area of ​​the first empty space and the area of ​​the second empty space, specifically as follows: The fullness acquisition module calculates the fullness of each corner container using the following formula, based on the area of ​​the first empty space and the area of ​​the corresponding second empty space: R ci_full_ratio =1_max(0,min(A ci_e / A ci_all ,1)) Among them, R ci_full_ratio A represents the fullness of the corner container. ci_e A is the area of ​​the first empty space in each of the corner containers. ci_all The area of ​​the second empty space corresponding to each corner container.

16. The fullness detection device based on target segmentation as described in claim 9, characterized in that, The fullness detection device also includes an information feedback module, specifically defined as follows: The information feedback module uses the product inventory detected by the terminal or server as an estimate of the total product inventory. When the product inventory on the terminal or server is lower than a preset threshold, the saturation level is transmitted to the business system so that the user can respond to the saturation level transmitted by the business system and the obtained saturation level is returned to the caller in the form of service parameters.

Images