Processing device, processing method, and processing program for image including container

Abstract

The objective of the present invention is to obtain a preferred (improved) processing device, processing method, and processing program for an image including a container. Provided is a processing device for an image including a container (40) having an upper surface opening part (41), a lower surface bottom part (42), a lateral surface connection part (43) that connects the upper surface opening part and the lower surface bottom part, an upper surface sealing part (44) that seals the upper surface opening part, an upper surface side display part (45) provided on the upper surface sealing part, and a lateral surface side display part (46) provided on the lateral surface connection part, the processing device comprising: a first straight line acquisition unit (25X) for acquiring a first straight line based on the upper surface side display part from the image including the container; a second straight line acquisition unit (25Y) for acquiring a second straight line based on the lateral surface side display part; and an angle acquisition unit (25Z) for acquiring an angle formed by the first straight line and the second straight line.

Description

Processing device, processing method, and processing program for images including containers

[0001] The present disclosure relates to a processing device, a processing method, and a processing program for an image including a container.

[0002] Patent Document 1 describes a drunk driving prevention device that includes an individual identification information reading means, a position estimation means, and a drunk driving prevention control means. The individual identification information reading means is capable of reading individual identification information from an alcoholic beverage container present in the vehicle cabin. The position estimation means estimates the position of the alcoholic beverage within the vehicle cabin based on the output of the individual identification information reading means. The drunk driving prevention control means performs predetermined control to prevent drunk driving based on the position of the alcoholic beverage estimated by the position estimation means. Patent Document 1 also exemplifies an RFID reader as a specific example of a device that reads the individual identification information of an alcoholic beverage. Patent Document 1 further describes that the device that reads the individual identification information of an alcoholic beverage is not limited to one that uses wireless communication, and that it may also be one that captures an image of a barcode or the like and reads the individual identification information of the alcoholic beverage using image recognition, for example.

[0003] Patent Document 2 describes a notification control system including a user information receiving means, a store information receiving means, a matching means, and a purchase information notifying means. The user information receiving means receives user information about beverages posted by users. The store information receiving means receives store information about beverage purchases at physical or virtual stores. The matching means matches the user information with store information regarding beverage attributes. The purchase information notifying means notifies the user of purchase information based on the user information and store information matched by the matching means. Patent Document 1 also describes that the matching means performs image recognition on image data of the user information to identify beverage attributes, and that known image recognition technology (such as OCR technology or shape determination technology) may be used to perform image recognition on the image data of the user information to identify beverage attributes.

[0004] JP 2009-258855 A JP 2017-091343 A

[0005] However, the prior art including Patent Documents 1 and 2 leaves room for improvement in terms of processing images including suitable (improved) containers.

[0006] Therefore, one of the objects of the present disclosure is to provide a processing device, a processing method, and a processing program for an image including a suitable (improved) container.

[0007] An image processing device including a container according to one aspect of the present disclosure is an image processing device including a container having a top opening portion, a bottom bottom portion, a side connection portion connecting the top opening portion and the bottom bottom portion, a top sealing portion sealing the top opening portion, a top display portion provided on the top sealing portion, and a side display portion provided on the side connection portion, and is characterized by having an image acquisition unit that acquires an image including the container that is captured by an imaging unit of a user terminal and transmitted by an image transmission unit of the user terminal, a first line acquisition unit that acquires a first straight line from the image based on the top display portion, a second line acquisition unit that acquires a second straight line from the image based on the side display portion, and an angle acquisition unit that acquires the angle between the first line and the second line.

[0008] According to one aspect of the present disclosure, it is possible to obtain a processing device, a processing method, and a processing program for an image including a suitable (improved) container.

[0009] 1 is a diagram showing an example of the configuration of a system including a processing device for an image including a container. FIG. 1 is a diagram showing an example of the configuration of a user terminal. FIG. 1 is a diagram showing an example of an image including a container. FIG. 2 is a diagram showing an example of the concept of a container image analysis application service based on an image including a container. FIG. 2 is a diagram showing an example of the configuration of an image processing device. FIG. 3 is a diagram showing an example of the configuration of an angle calculation unit. FIG. 4 is a diagram showing an example of a first line acquisition process by a first line acquisition unit. FIG. 5 is a diagram showing an example of a second line acquisition process by a second line acquisition unit. FIG. 6 is a diagram showing the angle formed by the first line and the second line acquired by the angle acquisition unit. FIG. 7 is a diagram showing an example of the concept of a container image analysis application service based on an image including a container. FIG. 8 is a diagram showing a first example of the flow of a method for processing an image including a container. FIG. 9 is a diagram showing a second example of the flow of a method for processing an image including a container. FIG. 10 is a diagram showing an example of the functional configuration of a user terminal and a processing device for an image including a container. FIG. 11 is a diagram showing an example of the hardware configuration of a user terminal and a processing device for an image including a container.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0011] 1 is a diagram showing an example of the configuration of a system 1 including a processing device 20 for images including containers. The "processing device for images including containers" may be read as a "processing system for images including containers" (these may be synonymous). The method and program for processing images including containers are realized by causing a computer (CPU: Central Processing Unit) included in the system 1 (user terminal 10, processing device 20 for images including containers) to execute various processing steps.

[0012] As shown in FIG. 1, the system 1 includes a user terminal 10, a processing device for images including containers (hereinafter referred to as the "image processing device") 20, and a network 30 connecting the user terminal 10 and the image processing device 20. In FIG. 1, the user terminal 10 and the image processing device 20 are depicted as a simplified single configuration for the sake of convenience. In reality, the user terminal 10 and the image processing device 20 have more complex configurations than those depicted in FIG. 1, both in terms of hardware and software configuration. For example, the image processing device 20 may be configured to include multiple servers, databases, etc.

[0013] The system 1 may be understood as a configuration including a user terminal 10 and an image processing device 20, or as a configuration including only the image processing device 20 without the user terminal 10. In other words, the user terminal 10 is not an essential component of the system 1 (it is sufficient that an image including a container, which will be described later, is input to the image processing device 20).

[0014] The user terminal 10 is, for example, a mobile terminal (mobile communication terminal) such as a mobile phone, a smartphone, or a tablet terminal. In other words, the user terminal 10 in this specification is a communication device capable of performing various types of communication with the image processing device 20. The user terminal 10 may be, for example, a wired and / or wireless (e.g., Long Term Evolution (LTE) or 5 th The user terminal 10 may communicate with a network (such as the Internet) via a network such as 5G New Radio (5G NR) or Wi-Fi (registered trademark). The user terminal 10 executes a browser and various other applications (particularly a container image analysis application service described below) in response to user operations.

[0015] The image processing device 20 may be any communication device capable of performing various types of communication with the user terminal 10, and may be interpreted as a device, circuit, etc. In particular, the image processing device 20 performs various types of communication with the user terminal 10 to analyze an image including a container acquired (captured) by the user terminal 10 and provide various application services (container image analysis application services) based on the analysis results.

[0016] Fig. 2 is a diagram showing an example of the configuration of the user terminal 10. The user terminal 10 has an imaging unit (image acquisition unit) 11, an image transmission unit 12, and a container image analysis application unit 13. In addition to the components shown in Fig. 2, the user terminal 10 also has components that are installed in a general user terminal, such as a display, a speaker, and a microphone.

[0017] The imaging unit 11 is a camera system mounted on the user terminal 10, and captures (obtains) an image including a container (a container before opening). The image transmission unit 12 transmits the image including the container captured by the imaging unit 11 to the image processing device 20. The container image analysis application unit 13 executes various processes for utilizing a container image analysis application service based on the image including the container transmitted from the image transmission unit 12 to the image processing device 20.

[0018] For example, when the camera of the imaging unit 11 is activated, a photograph may be taken by determining whether the container in the image captured by the imaging element (CCD: Charge Coupled Device) is a predetermined brand. To achieve a more comfortable user experience (UX) than capturing an image separately, the container determination process may be divided. In the determination process, for example, the outer shape and design of the container may be analyzed to determine whether the image captured by the imaging element includes the logo of the predetermined brand or the top surface of the container. In this case, the probability (reliability score) that the container captured by the imaging element is a predetermined brand may be calculated, and imaging may be performed if the probability is equal to or greater than a predetermined value (e.g., 50% or greater). The predetermined value that determines whether or not imaging is performed is flexible, allowing for various design modifications. Of course, whether the container captured by the imaging element is a predetermined brand may also be detected using a 1 / 0 flag.

[0019] 3A, 3B, and 3C are diagrams showing examples of images including a container. As shown in FIGS. 3A to 3C, the container 40 has a top opening 41, a bottom bottom 42, a side connection portion 43 connecting the top opening 41 and the bottom bottom 42, and a top sealing portion 44 sealing the top opening 41. FIGS. 3A to 3C depict the container 40 before opening, with the top opening 41 sealed by the top sealing portion 44, and the top opening 41 hidden by the top sealing portion 44 (the leader line of the top opening 41 is depicted by a dashed line). Image processing in this embodiment targets images including the container 40 before opening. In this specification, the "shape of the top sealing portion 44 (a perfect circle or ellipse, as described below)" refers to the top surface of the top sealing portion 44 (a surface parallel or approximately parallel to the plane including the top opening 41).

[0020] The top sealing portion 44 is provided with a top display portion 45, which is simply depicted as "ABC" in the examples of Figures 3A to 3C. The side connection portion 43 is provided with a side display portion 46, which is simply depicted as "XYZ" in the examples of Figures 3A to 3C. For convenience of drawing, the top display portion 45 and the side display portion 46 are depicted in a normal font in Figures 3A to 3C, but in reality, the top display portion 45 and the side display portion 46 are distorted in images depending on the shape, curvature, etc. of the top sealing portion 44 and the side connection portion 43 of the container 40.

[0021] 3A shows, for example, a small lactic acid bacteria beverage container 40, with a top open portion 41 having a small diameter circle and a bottom bottom portion 42 having a large diameter circle, and a side connection portion 43 having a cylindrical shape that expands in diameter from the top open portion 41 toward the bottom bottom portion 42, then becomes a fixed diameter, then becomes depressed once, and then becomes a fixed diameter again. The top sealing portion 44 is made of a film member (a foil) that is placed over and adhered to the top open portion 41, and the top sealing portion 44 is removed to expose the top open portion 41 by pinching and peeling off the peripheral portion of the film member (the portion that hangs down and outside the top open portion 41).

[0022] 3B shows, for example, a container 40 for cup noodles (cup ramen, cup soba, cup udon, etc.), in which an upper open portion 41 is a large-diameter circle, a lower bottom portion 42 is a small-diameter circle, and a side connection portion 43 is a cylinder whose diameter decreases from the upper open portion 41 toward the lower bottom portion 42. The upper sealing portion 44 is made of a lid member (a flap) that is placed over and adhered to the upper open portion 41, and by pinching and peeling off the peripheral portion of the lid member (the portion that hangs down outside the upper open portion 41), the upper sealing portion 44 is removed and the upper open portion 41 is exposed.

[0023] 3C shows, for example, a PET bottle-type container 40 for beverages (mineral water, tea, soft drinks, etc.), in which the top open portion 41 is a small-diameter circle, the bottom bottom portion 42 is a large-diameter circle, and the side connection portion 43 is a cylinder whose diameter increases from the top open portion 41 toward the bottom bottom portion 42 before becoming a constant diameter. The top sealing portion 44 is made of a cap portion having a threaded portion that screws into a threaded portion formed on the periphery of the top open portion 41, and by gripping and turning the cap portion, the top sealing portion 44 is removed to expose the top open portion 41.

[0024] The shape of the container 40 depicted in Figures 3A to 3C is merely an example, and various other shapes of containers 40 are possible. The shapes and configurations of the top open portion 41, bottom bottom portion 42, side connection portion 43, and top sealing portion 44 are flexible, allowing for various design modifications. For example, the top open portion 41, bottom bottom portion 42, and top sealing portion 44 may be circular with the same or different diameters, or may have a shape other than circular (although a symmetrical shape that allows a center point to be defined is preferable). Furthermore, the side connection portion 43 may have a shape in which the center of the height direction is more bulging than the top or bottom, like a beer keg (a Western-style barrel with a bulging center for filling beer). The contents of the container 40 are flexible, allowing for various design modifications (it can contain any contents other than food or beverages).

[0025] 4A to 4C are diagrams illustrating an example of the concept of a container image analysis application service based on images containing containers. The container image analysis application service creates brand fans through the experience of capturing a container with a camera while providing benefits (value-added experiences). To receive the container image analysis application service, a user must confirm and agree to the terms of use and privacy policy via a user terminal 10 and then indicate their intention to participate. The container image analysis application service also includes settings such as the campaign name, application period, target products (e.g., container type and size), prizes, number of winners, and required points.

[0026] 4A, the image capturing unit 11 of the user terminal 10 captures (scans) an image of the container before the user eats or drinks from it, and transmits the captured image to the image processing device 20. Here, the image capturing and transmission of the container image in FIG. 4A may be limited in time or number of times (for example, once a day).

[0027] In Fig. 4B, when the image of the container shown in Fig. 4A is captured and transmitted a predetermined number of times, an automatic application for an original product (prize) is made. In the example of Fig. 4B, capturing and transmitting the image of the container shown in Fig. 4A is converted into one point, and an automatic application for an original product (prize) is made every time five points are accumulated. Here, the automatic application for every five points may be subject to time and number of times (for example, a maximum of ten times during the campaign period).

[0028] In Fig. 4C, if a user wins an automatic entry every 5 points, the user will win (be provided with) original goods. Fig. 4C shows an example of the original goods, which is a mobile device case (original smartphone case) with the container manufacturer XYZ and the product name ABC printed on it.

[0029] The above-described container image analysis application service is intended to promote sales of containers (related products), and therefore, by its nature, it is assumed (presumed) that the containers associated with the container images sent to the image processing device 20 are not identical. In other words, each of the predetermined points (e.g., 5 points) required for automatic application for original goods cannot be obtained by multiple images of the same container or images obtained from the Internet, but rather by images taken each time the user eats or drinks (before eating or drinking) the containers from which the contents are consumed. Furthermore, it is assumed (presumed) that the containers from which the user eats or drinks are purchased by, for example, the user, the user's family or housemates, or friends or colleagues who eat or drink the contents of the containers together with the user.

[0030] The processing device, processing method, and processing program for images including containers disclosed herein, from the perspective of solidifying the above assumptions and presuppositions, are equipped with components that can process (analyze) containers included in an image to determine whether the containers included in the image are the same or not, and can even lead to "proof of purchase" for the container. For example, smooth and reliable proof of purchase would make it easier to create loyalty programs (marketing measures in which companies award rewards to "good customers" who repeatedly use services or frequently purchase products) even for in-store purchases, and it would be possible to elevate digital sales promotion to a marketing customer relationship process (CRP) for increasing brand fan bases.

[0031] The following factors make it difficult to provide smooth and reliable proof of purchase: (1) Proof of purchase cannot be mistaken, considering the risk of complaints and fraud. (2) Postcards are difficult to distribute and are discarded each time, resulting in a huge number of serial numbers. (3) Serialized proof of purchases require processing on factory production lines, issuing barcodes, and printing stickers, resulting in a huge number of serial numbers. (4) Receipts are cumbersome and time-consuming, and are costly to verify and open up loopholes for fraud. In the past, there have been cases of organized receipt proof of purchase fraud (falsifying the date, folding receipts to create different payment methods, altering background colors, stacking multiple receipts) during sales promotion campaigns. (5) Electronic payment methods such as "XX Pay," which are less prone to errors, face problems due to payment restrictions, limited retailers, and a lack of flexibility due to being driven by platform providers (businesses that provide the infrastructure).

[0032] Thus, there has been a need for a system that allows a container manufacturer (in-house) to process (analyze) the containers in an image to determine whether the containers in the image are the same or different, and even to link them to proof of purchase for the container. Based on this insight, the present inventor has realized a processing device, processing method, and processing program for images containing containers that have such a system. This system shifts the concept of campaign proof of purchase from a premise of human wickedness to a premise of human goodness, and includes the flexibility to award slightly excessive points as long as obvious fraud or impropriety is detected. It is based on the possibility of so-called "pseudo proof of purchase," which does not require strict purchase determination.

[0033] 5 is a diagram showing an example of the configuration of the image processing device 20. The image processing device 20 has an image input unit (image acquisition unit) 21, a first determination unit 22, a second determination unit 23, a third determination unit 24, an angle calculation unit 25, an angle output unit 26, a comparison unit 27, a notification unit 28, and a bonus granting unit 29.

[0034] The image input unit (image acquisition unit) 21 inputs (acquires) an image including a container that has been captured by the imaging unit 11 of the user terminal 10 and transmitted from the image transmission unit 12. The image input unit (image acquisition unit) 21 inputs (acquires) multiple images from the same user terminal 10.

[0035] The first determination unit 22 analyzes the image input to the image input unit 21 and determines whether or not the container included in the image is a predetermined brand. This determination may be the same as the process used to determine whether or not to capture an image of a container using the imaging unit 11 of the user terminal 10. The first determination unit 22 may calculate a probability (reliability score) that the container included in the image input to the image input unit 21 is a predetermined brand.

[0036] The second determination unit 23 analyzes the image input to the image input unit 21 and determines whether the container included in the image has a predetermined capacity. For example, the second determination unit 23 may determine whether the container included in the image input to the image input unit 21 has a first capacity, a second capacity, or another capacity. The second determination unit 23 may calculate the probability (reliability score) of whether the container included in the image input to the image input unit 21 has the first capacity, the second capacity, or another capacity.

[0037] The third determination unit 24 analyzes the image input to the image input unit 21 and determines whether the image is an existing image lacking originality. The third determination unit 24 detects whether the image input to the image input unit 21 is an image picked up from the Internet (i.e., an image that is highly likely not taken by the user himself / herself via the user terminal 10). The third determination unit 24 uses template matching to compare the image with an Internet image sent to a backend (a part invisible to the user, such as a server-side or database system) and measure the similarity between the two. The third determination unit 24 may calculate the probability (reliability score) that the image input to the image input unit 21 is an existing image lacking originality.

[0038] The angle calculation unit 25 analyzes the image input to the image input unit 21 to determine a first line and a second line set based on the contour characteristic parts of the container, and calculates the angle between the first line and the second line. In the present disclosure, the first line and the second line are defined by using a top display unit 45 provided on the top sealing unit 44 and a side display unit 46 provided on the side connection unit 43 as contour characteristic parts of the container 40. This will be described in detail below.

[0039] 6 is a diagram showing an example of the configuration of the angle calculation unit 25. The angle calculation unit has a first line acquisition unit 25X, a second line acquisition unit 25Y, and an angle acquisition unit 25Z.

[0040] The first line acquisition unit 25X acquires a first line based on the upper surface display unit 45 provided on the upper surface sealing unit 44 from the image input to the image input unit 21 .

[0041] 7A, 7B, and 7C are diagrams showing an example of the first line acquisition process performed by the first line acquisition unit 25X.

[0042] As shown in FIG. 7A , the first line acquisition unit 25X corrects the top sealing portion 44, which is captured in an image input to the image input unit 21, from a non-circular shape (e.g., an ellipse) to a perfect circle. At this time, distortion of the top display unit 45 provided on the top sealing portion 44 is corrected. Note that if the container 40 is captured from directly above and the top sealing portion 44 in the image is a perfect circle, the correction process of FIG. 7A is not necessary. The correction process of FIG. 7A can also be performed after the first line acquisition and update processes of FIGS. 7B and 7C . For example, if the top sealing portion 44 is captured in an elliptical or approximately elliptical shape, the center point of the ellipse or approximately ellipse, which is the intersection of the major and minor axes, corresponds to the "center point of the top sealing portion 44."

[0043] As shown in Fig. 7A, the upper display unit 45 provided on the upper sealing unit 44 is configured to include the character string "ABC" repeatedly arranged in a predetermined column direction and row direction. In the example of Fig. 7B, the character string "ABC" is written in the left-right direction (column direction) in the figure, and the character string "ABC" is repeatedly written in the up-down direction (row direction) in the figure. A predetermined space is provided between the character strings "ABC" in the up-down direction (row direction).

[0044] 7B , the first line acquisition unit 25X acquires, as the first line, a line along the character string "ABC," more specifically, a line drawn by a dashed dotted line along the topmost character string "ABC," from the top display unit 45 provided on the top sealing unit 44. And / or, the first line acquisition unit 25X acquires, as the first line, a line along the boundary between adjacent character strings "ABC," more specifically, a line drawn by a dashed dotted line along the boundary between the bottommost character string "ABC" and the character string "ABC" above it from the top display unit 45 provided on the top sealing unit 44.

[0045] As shown in Fig. 7C , the first line acquisition unit 25X may acquire, as an updated version of the first line, a line (dashed line) along the character string "ABC" acquired in Fig. 7B or a line (dashed line) along the boundary between adjacent character strings "ABC" acquired in Fig. 7B, translated so that the line passes through the center point of the top sealing portion 44 (the center point of the perfect circle). The updated first line is depicted by a dashed line. Note that the update process of Fig. 7C may be omitted.

[0046] The first line acquisition unit 25X may vary the acquisition and update processes of the first line depending on the shape of the top sealing portion 44 reflected in the image input to the image input unit 21. For example, if the top sealing portion 44 reflected in the image is a perfect circle, the updated version of the first line may be acquired by translating a line (dash-dotted line) along the character string "ABC" or a line (dash-two dotted line) along the boundary between adjacent character strings "ABC" so that the line passes through the center point of the top sealing portion 44 (the center point of the perfect circle). On the other hand, if the top sealing portion 44 reflected in the image is an oval, the updated version of the first line may be acquired by translating a line (dash-dotted line) along the character string "ABC" or a line (dash-two dotted line) along the boundary between adjacent character strings "ABC" so that the line passes through the center point of the major axis or minor axis of the oval of the top sealing portion 44.

[0047] There is a degree of freedom in the specific manner in which the first line acquisition unit 25X acquires and updates the first line based on the top display unit 45, and various design modifications are possible. For example, a line along a character string included in the top display unit 45 or a line along a boundary line between adjacent character strings may be rotated in the same direction (e.g., clockwise or counterclockwise) by a predetermined angle (e.g., 90° or 45°) to be acquired as the first line. The first line may be acquired and updated according to a common (identical) rule based on the top display unit 45.

[0048] The second line acquisition unit 25Y acquires a second line based on the side display unit 46 provided on the side connection unit 43 from the image input to the image input unit 21 .

[0049] The side display unit 46 includes a reference point provided on the side connection unit 43. This "reference point" is set, for example, as a part of the character string "XYZ" provided on the side connection unit 43 (side display unit 46). Here, the center (center point) of the letter "Y" is set as the "reference point provided on the side connection unit 43 (side display unit 46)." As will be described in more detail below, the "reference point provided on the side connection unit 43 (side display unit 46)" is extracted and determined using segmentation and keypoint extraction (particularly keypoint extraction), with a point on the logo on the side of the container (e.g., the center of a part of the character string) as the target. This also applies to the extraction and determination of the character string "ABC" provided on the top display unit 45 and the straight line along the character string.

[0050] FIG. 8 is a diagram illustrating an example of a second line acquisition process performed by the second line acquisition unit 25Y. As illustrated in FIG. 8, the center point A of the top sealing portion 44 is acquired (set, extracted). Also, a reference point provided on the side connection portion 43 (side display portion 46), i.e., point B where a line extending perpendicularly from the center (center point) of the letter "Y" toward a plane including the top sealing portion 44 intersects with the plane including the top sealing portion 44, is acquired (set, extracted). The line connecting point A and point B is then acquired as the second line. In this manner, the second line acquisition unit 25Y acquires, from the image input to the image input unit 21, a line connecting the center point A of the top sealing portion 44 and point B where a line extending perpendicularly from the reference point of the side display portion 46 toward a plane including the top sealing portion 44 intersects with the plane including the top sealing portion 44.

[0051] 7A to 7C and 8, the top surface of the container (top sealing portion 44) is circular (approximately circular) in plan view. Therefore, when capturing an image of the container, if the image is captured from directly above the container, the top surface of the container (top sealing portion 44) will appear circular (approximately circular). However, if the image is captured from even a slight angle, the top surface of the container (top sealing portion 44) will appear non-circular, and the latter case is likely to be the most common. Therefore, the first line acquisition unit 25X may correct the top surface of the container (top sealing portion 44) captured in the image from a non-circular shape to a perfect circle before acquiring the first line, and the second line acquisition unit 25Y may correct the top surface of the container (top sealing portion 44) captured in the image from a non-circular shape to a perfect circle before acquiring the second line.

[0052] Angle acquisition unit 25Z acquires the angle formed by the first line acquired by first line acquisition unit 25X and the second line acquired by second line acquisition unit 25Y. The first line acquired in Figures 7A to 7C and the second line acquired in Figure 8 are located on the same plane along the top surface of top sealing portion 44, so it is possible to acquire the angle formed by the first line and the second line. Figure 9 is a diagram showing the angle θ formed by the first line and the second line acquired by angle acquisition unit 25Z.

[0053] The angle calculation by the angle calculation unit 25 (first line acquisition unit 25X, second line acquisition unit 25Y, angle acquisition unit 25Z) is performed through three processes: segmentation, keypoint extraction, and angle calculation through geometric processing. That is, when an image including a container is sent to the backend (a part invisible to the user, such as a server-side or database system), the three processes of segmentation, keypoint extraction, and angle calculation through geometric processing are performed, resulting in the determination of a first line and a second line, and the angle between the first line and the second line being obtained. In this way, the first line acquisition unit 25X and the second line acquisition unit 25Y acquire the first line and the second line based on the segmentation and keypoint extraction.

[0054] Segmentation is a model that estimates what is in an image for each pixel, and additional learning can be performed by creating original training data using a model architecture used in general object detection. Segmentation determines, on a pixel-by-pixel basis, the area of ​​the target image that is the top surface of the container.

[0055] Keypoint extraction is a model that detects the coordinates of points that are characteristic of an object in an image, and learning based on the model is possible by setting new keypoints for containers in the image. Keypoint extraction detects the coordinates of points in the target image that define (identify) the first and second lines. For example, the coordinates of the center point of the top sealing portion 44 and the plane including the top surface of the top sealing portion 44 are detected to define (identify) the first and second lines. Also, the coordinates of the character string "ABC" that defines (identify) the first line are detected. Also, the coordinates of the character string "XYZ" and the reference point in the character string (e.g., the center point of the letter "Y") that defines (identify) the second line are detected.

[0056] In angle calculation by geometric processing, the angle θ formed by the first line and the second line is calculated based on data detected by segmentation and keypoint extraction (see FIG. 9).

[0057] For example, segmentation takes an image that includes a container as input and determines the location of the top area of ​​the container on a pixel-by-pixel basis. Keypoint extraction takes an image that includes a container as input and identifies points to define (identify) the first and second lines. Angle calculation using geometric processing takes the results of segmentation and keypoint extraction as input and determines the first and second lines, and obtains the angle θ between the first and second lines.

[0058] When multiple images from the same user terminal 10 are input to the image input unit 21, the angle calculation unit 25 (first line acquisition unit 25X, second line acquisition unit 25Y, angle acquisition unit 25Z) calculates the angle θ between the first line and the second line for each of the multiple images.

[0059] The angle output unit 26 outputs the angle θ formed by the first line and the second line calculated by the angle calculation unit 25 (first line acquisition unit 25X, second line acquisition unit 25Y, angle acquisition unit 25Z). When multiple images from the same user terminal 10 are input to the image input unit 21 and the angle calculation unit 25 calculates the angle θ formed by the first line and the second line for the multiple images, the angle output unit 26 outputs the angle θ formed by the first line and the second line for the multiple images.

[0060] The comparison unit 27 compares a plurality of images from the same user terminal 10 to determine whether the angle θ formed by the first line and the second line is within a predetermined range.

[0061] During the manufacture of containers, the positional relationship between the top sealing portion (the top display portion, e.g., the character string "ABC") that seals the top open portion of the container and the side connection portion (the side display portion, e.g., the character string "XYZ") of the container is not uniformly manufactured, and the positional relationship between the two can be considered to be the degree of originality of each individual container. In other words, strictly speaking, the angle θ formed by the first line and the second line is almost always different for each individual container, and the possibility that the same container is included in multiple images can be determined based on this angle θ. For example, if the angle θ formed by the first line and the second line for containers included in multiple images is significantly different, there is a high possibility that the containers included in the multiple images are different. However, if the angle θ formed by the first line and the second line is the same or almost the same, there is a high possibility that the containers included in the multiple images are the same.

[0062] For example, the comparison unit 27 determines the angle θ between the first line and the second line in any one of the reference images as a reference angle θ ｓｔａｎｄａｒｄ and the angle θ formed by the first line and the second line in the other reference images of the plurality of images is set as the reference angle θ ｒｅｆｅｒｅｎｃｅ Then, the reference angle θ ｓｔａｎｄａｒｄ and the reference angle θ ｒｅｆｅｒｅｎｃｅ is compared to see if it is within a predetermined range. For example, 0.95 × reference angle θ ｒｅｆｅｒｅｎｃｅ ≦Reference angle θ ｓｔａｎｄａｒｄ ≦1.05×reference angle θ ｒｅｆｅｒｅｎｃｅIf the conditional expression is not satisfied, it can be determined that the containers included in the reference image and the standard image are different. The comparison unit 27 may calculate a probability (reliability score) indicating whether the containers included in the multiple images (standard image and reference image) are the same.

[0063] When the angle θ between the first line and the second line for multiple images from the same user terminal 10 is within a predetermined range, i.e., when there is a high possibility that the containers included in the multiple images are the same, the notification unit 28 notifies the user terminal 10 of this fact. The notification method for the user terminal 10 can be, for example, a display on a display and / or a voice from a speaker. In this case, the notification unit 28 may notify the user that the multiple images are not eligible for a bonus by the bonus granting unit 29, which will be described later.

[0064] In addition, if the area of ​​the container in the image including the container input to the image input unit 21 is less than a predetermined range, the notification unit 28 may issue an error notification, stating that it is unable to calculate (obtain) the angle θ formed by the first line and the second line.

[0065] When the angle θ formed by the first line and the second line for multiple images from the same user terminal 10 is not within a predetermined range, the bonus awarding unit 29 determines that the containers included in the multiple images are different, and awards a bonus according to the number of multiple images to the user terminal 10. For example, each image that includes a container may be converted into one point, and when the number of points reaches five, corresponding to five images, an automatic application may be made, and if the user wins the lottery, they may win original goods (see FIGS. 4B and 4C ).

[0066] FIG. 10 illustrates an example of the concept of a container image analysis application service based on an image containing a container. As shown in FIG. 10, an image containing a container is sent from an application on a user terminal 10 to an AI (artificial intelligence) in an image processing device 20. The AI ​​in the image processing device 20 assigns container information to the uploaded image. More specifically, information such as the angle θ between the first and second lines for each container, the similarity with images on the Internet, the angle θ between the first and second lines for a container most recently uploaded by the same user, and product information (e.g., the container's capacity) can be obtained. The results of the AI ​​in the image processing device 20 are passed as numerical values ​​to a campaign system, and the campaign system is configured to regulate point and bonus awarding.

[0067] FIG. 11 is a diagram showing a first example of the flow of a method for processing an image including a container.

[0068] In step ST1, the camera serving as the imaging unit 11 of the user terminal 10 is activated. In step ST2, an image including a container is captured (obtained) by the imaging unit 11 of the user terminal 10. When the camera is activated and recognizes a logo or a specific part of the container (for example, characters or a pattern indicating the product name) (when the probability that the container is the target container exceeds a predetermined value), the shutter is automatically released to capture an image including the container.

[0069] In step ST3, the second determination unit 23 of the image processing device 20 determines whether the container has a predetermined capacity. This determination may be made, for example, by distinguishing the capacity based on the overall shape of the container, or by using the product ID (JAN code) attached to the container. In step ST4, the third determination unit 24 of the image processing device 20 confirms that the image including the container is not an existing image (an image picked up from the Internet) that lacks originality.

[0070] In step ST5, the angle calculation unit 25 (first line acquisition unit 25X, second line acquisition unit 25Y, angle acquisition unit 25Z) of the image processing device 20 calculates the angle between the first line and the second line for a container included in the captured image. In step ST6, the comparison unit 27 of the image processing device 20 compares the angle between the first line and the second line of a container included in an image most recently registered by a user with the same ID (the same user) with the angle calculated in step ST5, by checking against a database, and determines the degree of match. In step ST7, a final determination is made as to whether the containers included in the multiple images whose angles have been compared and determined in step ST6 are the same container or different containers.

[0071] Figures 12A, 12B, 12C and 12D show a second example of a method flow for processing an image containing a container, which depicts a container of the type illustrated in Figure 3A.

[0072] In FIG. 12A, filter 1 using a camera AI model determines whether the container is a specified brand. In FIG. 12B, filter 2 using a back-end AI model (object detection) determines whether the container is captured at a certain size and the "XYZ" character string on the side display. In FIG. 12C, filter 3 using a back-end AI model (keypoint detection) determines the "ABC" character string on the top display and determines the reference point (e.g., the center of the central "Y") of the "XYZ" character string on the side display. In FIG. 12D, filter 4 using a back-end AI model (segmentation) performs ellipse detection for angle calculation from the top sealing portion of the container and obtains the angle between the first line obtained based on the "ABC" character string on the top display and the second line obtained based on the "XYZ" character string on the side display.

[0073] In this manner, in the image processing device for images including containers according to the present disclosure, the first line acquisition unit acquires a first line based on the top display unit from the image including the container, the second line acquisition unit acquires a second line based on the side display unit from the image including the container, and the angle acquisition unit acquires the angle between the first line and the second line. This enables the processing of images including suitable (improved) containers. For example, analysis of the containers included in the image can be performed to determine whether the containers included in the image are the same, and can even be used to provide proof of purchase for the container. Furthermore, the system is flexible, convenient, and can achieve a certain level of accuracy.

[0074] Furthermore, the image processing device for containers according to the present disclosure maintains a probability of whether an image contains a specific container or whether it is the same container, ensuring flexibility in determining whether the image is correct or incorrect. Furthermore, detailed feedback of metadata surrounding the determination allows for flexible changes in campaign design. Furthermore, analysis processing is performed on the application server side (image processing device 20) and container recognition is performed on the application side (user terminal 10), enabling a user experience (UX) such as a real-time feel. Furthermore, implementation is easy, as it can be implemented using, for example, JavaScript (registered trademark) running on HTML.

[0075] Fig. 13 is a diagram showing an example of the functional configuration of a user terminal 10 and an image processing device 20 according to an embodiment. As shown in Fig. 13, the user terminal 10 and the image processing device 20 have a control unit 110, a storage unit 120, a communication unit 130, an input unit 140, and an output unit 150. Note that Fig. 13 shows an example of functional blocks of the user terminal 10 and the image processing device 20, and the user terminal 10 and the image processing device 20 may have other functional blocks not shown in Fig. 13. Furthermore, the configuration may not include some functional blocks.

[0076] The control unit 110 controls the user terminal 10 and the image processing device 20. The control unit 110 can be configured by a controller, a control circuit, or a control device that is described based on common understanding in the technical field related to the present invention.

[0077] The storage unit 120 stores information used by the user terminal 10 and the image processing device 20. The storage unit 120 can be configured, for example, by a memory, storage, storage device, or the like that is described based on common understanding in the technical field related to the present invention.

[0078] The communication unit 130 performs communication (e.g., mutual communication) between the user terminal 10 and the image processing device 20. The communication unit 130 can be configured by a transmitter / receiver, a transmission / reception circuit, or a transmission / reception device, which are described based on common understanding in the technical field related to the present invention. Note that the communication unit 130 may be configured by a transmission unit and a reception unit.

[0079] The input unit 140 accepts inputs from the user terminal 10 and the image processing device 20. The input unit 140 may also be connected to a predetermined device, storage medium, etc., and accept data input. The input unit 140 may output the input results to, for example, the control unit 110. The input unit 140 may be configured with input devices such as a keyboard, mouse, and buttons, input / output terminals, input / output circuits, etc., which are described based on common understanding in the technical field related to the present invention. The input unit 140 may also be configured as an integrated unit with a display unit (for example, a touch panel).

[0080] The output unit 150 performs output in the user terminal 10 and the image processing device 20. For example, the output unit 150 may be configured to include a display unit that displays images, an audio output unit that outputs audio, etc. The display unit may be configured, for example, by a display device such as a monitor or the like that is described based on common understanding in the technical field related to the present invention. Furthermore, the audio output unit may be configured by an output device such as a speaker that is described based on common understanding in the technical field related to the present invention.

[0081] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by a single physically coupled device, or may be realized by two or more physically separate devices connected by wire or wirelessly.

[0082] For example, the user terminal 10 and the image processing device 20 according to an embodiment of the present disclosure may function as a computer that processes the image processing method and processing program for an image including a container according to the present disclosure. Fig. 15 is a diagram illustrating an example of the hardware configuration of the user terminal 10 and the image processing device 20 according to an embodiment. The above-described user terminal 10 and the image processing device 20 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0083] In this disclosure, the terms apparatus, circuit, device, unit, server, etc. may be interchangeable. The hardware configuration of the user terminal 10 and the image processing device 20 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.

[0084] For example, although only one processor 1001 is shown, there may be multiple processors. Furthermore, processing may be performed by one processor, or processing may be performed by two or more processors simultaneously, serially, or in other ways. Furthermore, processor 1001 may be implemented by one or more chips.

[0085] Each function of the user terminal 10 and the image processing device 20 is realized by loading specified software (programs) onto hardware such as the processor 1001, memory 1002, etc., so that the processor 1001 performs calculations and controls communication via the communication device 1004, reading and / or writing of data in the memory 1002 and storage 1003, etc.

[0086] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. Note that each unit, such as the control unit 110 described above, may be realized by the processor 1001.

[0087] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, the control unit 110 may be implemented by a control program stored in the memory 1002 and executed by the processor 1001, and similar implementations may be made for other functional blocks.

[0088] The memory 1002 is a computer-readable recording medium and may be configured by at least one of, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically programmable read-only memory (EEPROM), a random access memory (RAM), or other suitable storage medium. The memory 1002 may also be referred to as a register, a cache, a main memory, or the like. The memory 1002 may store executable programs (program codes), software modules, and the like for implementing a method according to an embodiment.

[0089] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, a flexible disk, a floppy (registered trademark) disk, a magneto-optical disk (e.g., a compact disk (CD-ROM (Compact Disc ROM)), a digital versatile disk, a Blu-ray (registered trademark) disk), a removable disk, a hard disk drive, a smart card, a flash memory device (e.g., a card, a stick, a key drive), a magnetic stripe, a database, a server, or other suitable storage medium. Storage 1003 may also be called an auxiliary storage device. Note that the above-mentioned storage unit 120 may be realized by memory 1002 and / or storage 1003.

[0090] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may include a SIM card. Note that the above-mentioned communication unit 130 may be realized by the communication device 1004.

[0091] The input device 1005 is an input device (e.g., a keyboard, a mouse, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one unit (e.g., a touch panel). The above-mentioned input unit 140 and output unit 150 may be realized by the input device 1005 and the output device 1006, respectively.

[0092] Furthermore, each device such as the processor 1001 and the memory 1002 is connected by a bus 1008 for communicating information. The bus 1008 may be configured as a single bus or may be configured as different buses between the devices.

[0093] The user terminal 10 and the image processing device 20 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these pieces of hardware.

[0094] (Modifications) Note that terms explained in this disclosure and / or terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings.

[0095] The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or other corresponding information. Furthermore, the names used for parameters, etc. in this disclosure are not limiting in any way.

[0096] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0097] Information, signals, etc. may be input / output via multiple network nodes. The input / output information, signals, etc. may be stored in a specific location (e.g., memory) or may be managed using a table. The input / output information, signals, etc. may be overwritten, updated, or added. Output information, signals, etc. may be deleted. Input information, signals, etc. may be transmitted to another device.

[0098] Furthermore, notification of specified information (e.g., notification that "it is X") is not limited to explicit notification, but may be made implicitly (e.g., by not notifying the specified information or by notifying other information).

[0099] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0100] Software, instructions, information, etc. may also be transmitted and received via a transmission medium and / or signal waveform. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0101] As used in this disclosure, the terms "system" and "network" may be used interchangeably.

[0102] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Furthermore, the order of the processing procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0103] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0104] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0105] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0106] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0107] Although the invention according to the present disclosure has been described in detail above, it is clear to those skilled in the art that the invention according to the present disclosure is not limited to the embodiments described in the present disclosure. The invention according to the present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the invention as defined by the description of the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not impose any limiting meaning on the invention according to the present disclosure.

[0108] This application is based on Japanese Patent Application No. 2023-149680, filed September 15, 2023, the contents of which are incorporated herein in their entirety.

Claims

1. An image processing device including a container having a top opening portion, a bottom portion, a side connecting portion connecting the top opening portion and the bottom portion, a top sealing portion sealing the top opening portion, a top display portion provided on the top sealing portion, and a side display portion provided on the side connecting portion, characterized in having an image acquisition unit that acquires an image including the container captured by an imaging unit of a user terminal and transmitted by an image transmission unit of the user terminal, a first line acquisition unit that acquires a first line from the image based on the top display portion, a second line acquisition unit that acquires a second line from the image based on the side display portion, and an angle acquisition unit that acquires the angle between the first line and the second line.

2. The image processing device including a container as described in claim 1, characterized in that: the top display unit is configured to include a character string provided on the top sealing unit; the side display unit is configured to include a reference point provided on the side connection unit; the first straight line acquisition unit acquires as the first straight line a straight line along the character string or a straight line along the boundary line between adjacent character strings; and the second straight line acquisition unit acquires as the second straight line a straight line connecting a center point of the top sealing unit and a point at which a straight line extending perpendicularly from the reference point of the side display unit toward a plane including the top sealing unit intersects with the plane including the top sealing unit.

3. The image processing device including a container as described in claim 2, characterized in that the first straight line acquisition unit acquires, as an updated version of the first straight line, a straight line along the character string or a straight line along the boundary between adjacent character strings, translated so that it passes through the center point of the top sealing portion.

4. The image processing device including a container as described in claim 2, characterized in that the first straight line acquisition unit acquires, when the top sealing portion captured in the image is a perfect circle, a straight line along the character string or a straight line along the boundary between adjacent character strings, translated to pass through the center point of the perfect circle, as an updated version of the first straight line, and when the top sealing portion captured in the image is an ellipse, acquires, when the top sealing portion captured in the image is an ellipse, a straight line along the character string or a straight line along the boundary between adjacent character strings, translated to pass through the center point of the major axis or minor axis of the ellipse, as an updated version of the first straight line.

5. An image processing device including a container as described in claim 1 or claim 2, characterized in that the first straight line acquisition unit and the second straight line acquisition unit acquire the first straight line and the second straight line after correcting the top sealing portion captured in the image from a non-circular shape to a perfect circle.

6. The processing device for an image including a container as described in claim 1 or claim 2, characterized in that the first line acquisition unit and the second line acquisition unit acquire the first line and the second line based on segmentation and key point extraction.

7. A method for processing an image including a container having a top open portion, a bottom portion, a side connection portion connecting the top open portion and the bottom portion, a top sealing portion sealing the top open portion, a top display portion provided on the top sealing portion, and a side display portion provided on the side connection portion, characterized in that the method for processing an image including a container includes the following steps, which are executed by a computer: acquiring an image including the container that is captured by an imaging portion of a user terminal and transmitted by an image transmission portion of the user terminal; acquiring from the image a first straight line based on the top display portion; acquiring from the image a second straight line based on the side display portion; and acquiring the angle between the first straight line and the second straight line.

8. A processing program for an image including a container having a top opening portion, a bottom portion, a side connecting portion connecting the top opening portion and the bottom portion, a top sealing portion sealing the top opening portion, a top display portion provided on the top sealing portion, and a side display portion provided on the side connecting portion, the processing program for an image including a container comprising: a step of acquiring an image including the container captured by an imaging portion of a user terminal and transmitted by an image transmitting portion of the user terminal; a step of acquiring from the image a first straight line based on the top display portion; a step of acquiring from the image a second straight line based on the side display portion; and a step of acquiring an angle between the first straight line and the second straight line.

Images