Transmission device, reception device and information processing method

Abstract

A transmission device according to the present technology includes: an image acquisition unit that acquires a first shelf image captured with respect to a goods shelf on which goods are displayed at a first time, and a second shelf image captured with respect to the goods shelf at a second time later than the first time; a region recognition unit that recognizes a difference region between the first shelf image and the second shelf image; a cut-out processing unit that cuts out a partial image from the second shelf image such that the difference region is included therein; and a transmission processing unit that transmits the first shelf image and the partial image to a receiving device.

Description

Technical Field

[0001] The present technology relates to the technical field of a transmitting device that transmits a captured image of a product and a receiving device that receives the image. Background Art

[0002] Retail stores are increasingly introducing systems for managing products in order to simplify employees' work and to perform operations appropriately.

[0003] For example, in the following Patent Document 1, an image of a display shelf is captured to recognize items displayed on the shelf and grasp the status of out-of-stock items.

[0004] Such a function may be provided by a server device such as a cloud server, and when the method disclosed in Patent Document 1 is used, image recognition processing for detecting the state of out-of-stock items in the server device by sending a captured image to the server device is performed.

[0005] Reference List

[0006] Patent Literature

[0007] Patent document 1: JP 2020-126679 A. Summary of the invention

[0008] Technical issues

[0009] However, when considering that the server device provides the same function for multiple stores, there is a problem that the communication volume between the store terminal (or camera) and the server device increases when the captured image is sent to the server device. There is also a problem that the processing burden on the server device increases, resulting in higher power consumption.

[0010] The present technology is devised in view of such problems, and an object thereof is to appropriately perform processing for identifying the status of a product while reducing the burden on a server device.

[0011] Solution to the problem

[0012] The sending device according to the present technology includes: an image acquisition unit, which acquires a first shelf image captured at a first time for a product shelf on which products are displayed and a second shelf image captured at a second time after the first time for the product shelf; an area designation unit, which designates a difference area between the first shelf image and the second shelf image, a cutout processing unit, which cuts out a partial image from the second shelf image so as to include the difference area; and a sending processing unit, which sends the first shelf image and the partial image to a receiving device.

[0013] Therefore, the transmitting device transmits the image of the product shelf to the receiving device so as to issue various alarms according to the status of the products displayed on the product shelf. Then, the receiving device performs image recognition processing using the received image to recognize the out-of-stock status, inappropriate display position status, etc. of the product. At this time, the transmitted data is reduced by transmitting only the difference area between the first shelf image and the second shelf image. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a diagram showing an example of the configuration of the information processing system in the first embodiment.

[0015] Figure 2 is a schematic diagram showing data sent from a camera to a cloud server.

[0016] Figure 3 is a block diagram showing an example of the internal configuration of the camera in the first embodiment.

[0017] Figure 4 is a block diagram showing an example of the hardware configuration of an information processing device.

[0018] Figure 5 is a block diagram showing a functional configuration of a signal processing unit of the camera in the first embodiment.

[0019] Figure 6 is a diagram showing a difference area of ​​a processing target shelf image.

[0020] Figure 7 is a diagram showing display space on a product shelf.

[0021] Figure 8 is a diagram showing a partial image cut out to include a difference area.

[0022] Fig. 9 It is a block diagram showing the functional configuration of the CPU of the cloud server in the first embodiment.

[0023] Fig.10 is a diagram showing the timing of capturing and receiving a reference shelf image and processing a target shelf image in time series.

[0024] Fig.11 is a diagram showing a status of replenishing a product.

[0025] Fig.12 is a diagram showing an out-of-stock status of a product.

[0026] Fig.13 is a diagram showing a supply shortage state of a product.

[0027] Fig.14 It is a diagram showing an improper posture state of a product.

[0028] Fig.15 is a diagram showing an inappropriate display position state of a product.

[0029] Fig.16 is a diagram showing an inappropriate price tag state of a product.

[0030] Fig.17 is a diagram illustrating an example of data regarding a product stored in a database.

[0031] Fig.18 : is a diagram showing an example of a counting result of a notification process.

[0032] Fig.19 is a flowchart showing an example of processing performed by the camera before the store opens in the first embodiment.

[0033] Fig. 20 is a flowchart showing an example of processing performed by the camera after the store opens in the first embodiment.

[0034] Fig.21 is with Fig. 22 Flowcharts showing an example of processing executed by the cloud server in the first embodiment are shown together.

[0035] Fig. 22 is with Fig.21 1 and 1. A flowchart showing an example of processing performed by the cloud server in the first embodiment, and showing the following Fig.21 Flowchart of the processing performed thereafter.

[0036] Fig.23 is a flowchart illustrating an example of processing performed by the cloud server to optimize the imaging interval according to the number of notifications.

[0037] Fig.24 is a flowchart illustrating an example of processing performed by a camera to optimize the imaging interval.

[0038] Fig.25 is a block diagram showing an example of the internal configuration of a camera in the second embodiment.

[0039] Fig.26 : is a block diagram showing the functional configuration of the AI ​​image processing unit of the camera in the second embodiment.

[0040] Fig. 27 is a block diagram showing an example of the internal configuration of a camera in the third embodiment.

[0041] Fig.28 : is a block diagram showing the functional configuration of the AI ​​image processing unit of the camera in the third embodiment.

[0042] Fig.29 It is a block diagram showing the functional configuration of the CPU of the cloud server in the third embodiment.

[0043] Fig.30 : is a flowchart showing an example of processing performed by the camera before the store opens in the third embodiment.

[0044] Fig.31 is with Fig.32 A flowchart showing an example of processing performed by the camera after the store opens in the second embodiment is shown together.

[0045] Fig.32 is with Fig.31 1 and ... Fig.31 Flowchart of the processing performed thereafter.

[0046] Fig.33 is a flowchart showing an example of processing executed by the CPU of the cloud server in the third embodiment.

[0047] Fig.34 is a diagram showing an example of the configuration of an information processing system in the fourth embodiment.

[0048] Fig.35 is a diagram showing an example of a real-time inventory status screen in a management tool provided by a cloud server.

[0049] Fig.36 is a diagram illustrating an example of a real-time stock status screen in a management tool provided by a cloud server, and is a diagram illustrating a state in which a label indicating a stock status is displayed.

[0050] Fig.37 is a diagram showing an example of a setting screen in a management tool provided by a cloud server.

[0051] Fig.38 is a diagram showing an example of a product name management screen in a management tool provided by a cloud server.

[0052] Fig.39 is a diagram showing a functional configuration of a CPU of a cloud server that provides a management tool.

[0053] Fig.40 is a diagram showing each device registering and downloading an AI model and an AI application via a market function provided by a cloud-side information processing device.

[0054] Fig.41is a diagram illustrating an example of the flow of processing performed by each device when an AI model or an AI application is registered and downloaded via a market function.

[0055] Fig.42 is a diagram showing an example of the flow of processing performed by each device when an AI application or an AI model is deployed.

[0056] Fig.43 is a diagram showing a connection mode between a cloud-side information processing device and an edge-side information processing device.

[0057] Fig.44 It is a functional block diagram of a cloud-side information processing device.

[0058] Fig.45 is a diagram illustrating an example of a configuration of an image sensor.

[0059] Fig.46 is a block diagram showing the software configuration of the camera.

[0060] Fig.47 is a block diagram illustrating an operating environment of a container when container technology is used.

[0061] Fig.48 It is a diagram showing the flow of processing in other descriptions.

[0062] Fig.49 is a diagram showing an example of a login screen for logging into a market.

[0063] Fig.50 is a diagram showing an example of a developer screen presented to each developer who uses a marketplace.

[0064] Fig.51 is a diagram showing an example of a user screen presented to an application user who uses a market. DETAILED DESCRIPTION

[0065] Hereinafter, an embodiment of an information processing device according to the present technology will be described in the following order with reference to the drawings.

[0066] <1. Overall configuration of information processing system>

[0067] <2. Camera Configuration>

[0068] <3. Hardware Configuration of Information Processing Device>

[0069] <4. Functional Configuration>

[0070] <5. Processing Flow>

[0071] <5-1. Processing Related to Acquisition of Reference Shelf Image>

[0072] <5-2. Processing Related to Acquisition of Target Shelf Image>

[0073] <5-3. Processing of Notification>

[0074] <5-4. Processing for Optimizing Imaging Intervals According to the Number of Notifications>

[0075] <5-5. Processing for Optimizing Imaging Interval>

[0076] <6. Second Embodiment>

[0077] <7. Third Embodiment>

[0078] <8. Fourth embodiment>

[0079] <9. User Interface>

[0080] <10. Registration of AI models and AI applications>

[0081] <11. System Functional Overview>

[0082] <12. Deployment of AI models and AI applications>

[0083] <13. Others>

[0084] <14. Example of market screen>

[0085] <15. Conclusion>

[0086] <16. This technology>

[0087] <1. Overall configuration of information processing system>

[0088] In this embodiment, the information processing system 100 is a system that recognizes the status of products displayed on shelves and issues various notifications such as alarms. The product manager performs appropriate work based on the notifications to maintain an appropriate display status of the product and suppress loss of product sales opportunities.

[0089] The following will refer to Figure 1 and other figures describe a first embodiment of an information processing system 100 .

[0090] like Figure 1 As shown, the information processing system 100 includes a cloud server 1, a user terminal 2, and a plurality of cameras 3. In this example, the cloud server 1, the user terminal 2, and the cameras 3 are configured to be able to communicate with each other via a network 6 such as the Internet.

[0091] The user terminal 2 is an information processing device assumed to be used by a user who receives a service using the information processing system 100. That is, in this example, the user terminal 2 is an information processing device used by a manager of products displayed on a product shelf.

[0092] Each camera 3 includes an image sensor such as a charge coupled device (CCD) type image sensor or a complementary metal oxide semiconductor (CMOS) type image sensor, and captures an image of a subject to obtain image data (captured image data) as digital data.

[0093] The sensor included in the camera 3 is, for example, an RGB sensor that captures an RGB image or a distance measurement sensor that outputs a distance image.

[0094] The camera 3 obtains a plurality of image data of product shelves on which products are displayed by imaging. These image data are referred to as "shelf images".

[0095] The shelf image captured by the camera 3 includes a reference shelf image and a shelf image to be compared. Here, the reference shelf image is, for example, a shelf image captured at a point in time when the display of the product is completed before the store opens, and is a shelf image obtained by imaging the state of the product before the state is changed. This shelf image is referred to as a "reference shelf image".

[0096] The shelf image used for comparison with the reference shelf image is a shelf image captured during business hours of the store, for example, a shelf image obtained by imaging a state where some products have been sold and the number of displayed items has been reduced. This shelf image is referred to as a "processing target shelf image".

[0097] The camera 3 compares the reference shelf image with the processing target shelf image and detects the difference between them. Then, a partial image is cut out from the shelf image so that the partial image includes a difference area, which is an area where the difference between the processing target shelf image and the reference shelf image is detected. That is, the partial image is an image that is a partial area of ​​the processing target shelf image and includes the difference area.

[0098] Camera 3 sends the reference shelf image and the partial image to cloud server 1 (see Figure 2 ). The camera 3 sends the position information specifying the position of the partial image in the processing target shelf image together with the partial image to the cloud server 1.

[0099] When a first shelf image is acquired during business hours, and then a second shelf image is acquired after a certain period of time has passed, the first shelf image is set as a reference shelf image, and the second shelf image is set as a processing target shelf image. Thus, a change in the state of a product within a certain period of time can be detected. Therefore, an image area that changes within a certain period of time can be cut out as a partial image, and only the minimum amount of image data required can be sent to the cloud server 1.

[0100] When the image transmitted to the cloud server 1 includes the face of a consumer or the like, the camera 3 performs a privacy mask process and then transmits the image.

[0101] In this example, the processing in camera 3 is implemented as rule-based signal processing.

[0102] The cloud server 1 receives the reference shelf image and the partial image from the camera 3. The cloud server 1 cuts out a region corresponding to the partial image in the reference shelf image as a corresponding image, and compares the partial image with the corresponding image to specify the state of a product captured in the partial image (hereinafter, referred to as a "target product").

[0103] The state of the target product specified in the cloud server 1 is, for example, an out-of-stock state, a supply shortage state, a state in which the display position is inappropriate, a state in which the posture is inappropriate, or the like.

[0104] When the cloud server 1 specifies such a product status, an information notification (including an alarm) corresponding to the product status is notified to the product administrator.

[0105] The product status designation processing in the cloud server 1 is performed, for example, using artificial intelligence (AI). Specifically, a partial image and a corresponding image are input to an AI model to obtain information about the product status. The product status is output from the AI ​​model as label information, etc. In this way, image processing performed using an AI model and its associated program is referred to as "AI image processing".

[0106] The cloud server 1 and the user terminal 2 are respectively configured as information processing devices equipped with a microcomputer including a central processing unit (CPU), a read only memory (ROM), and a random access memory (RAM).

[0107] <2. Camera Configuration>

[0108] Figure 3 is a block diagram showing an example of the internal configuration of the camera 3 .

[0109] As shown in the figure, the camera 3 includes an imaging optical system 31, an optical system driving unit 32, an image sensor IS, a control unit 33, a memory unit 34, a communication unit 35, and a signal processing unit 36. The image sensor IS, the control unit 33, the memory unit 34, the communication unit 35, and the signal processing unit 36 ​​are connected to each other via a bus 37, and can perform data communication with each other.

[0110] The imaging optical system 31 includes lenses such as a cover lens, a zoom lens, and a focus lens, and an aperture mechanism. Light (incident light) from a subject is guided by the imaging optical system 31 and focused on a light receiving surface of the image sensor IS.

[0111] The optical system driving unit 32 collectively refers to a driving unit for the zoom lens, focus lens and aperture mechanism of the imaging optical system 31. Specifically, the optical system driving unit 32 includes actuators for driving the zoom lens, focus lens and aperture mechanism respectively and a driving circuit for the actuators.

[0112] The control unit 33 is configured to include, for example, a microcomputer including a CPU, a ROM, and a RAM, and the CPU performs overall control of the camera 3 by executing various processes according to a program stored in the ROM or a program loaded into the RAM.

[0113] Furthermore, the control unit 33 issues instructions to the optical system driving unit 32 to drive the zoom lens, focus lens, diaphragm mechanism, etc. In response to these driving instructions, the optical system driving unit 32 performs movement of the focus lens and zoom lens, opening and closing of diaphragm blades of the diaphragm mechanism, etc.

[0114] The control unit 33 also controls writing and reading of various data to and from the memory unit 34 .

[0115] The memory unit 34 is a nonvolatile storage device such as a hard disk drive (HDD) or a flash memory device, and serves as a storage destination (recording destination) of image data output from the image sensor IS.

[0116] Furthermore, the control unit 33 performs various data communications with external devices via the communication unit 35. The communication unit 35 in this example is configured to be able to communicate with at least Figure 1 The cloud server 1 shown performs data communication.

[0117] The signal processing unit 36 ​​is composed of, for example, a field programmable gate array (FPGA), and performs the above-mentioned processing for comparing the reference shelf image with the processing target shelf image, processing for cutting out a partial image, processing for specifying position information of a partial image in the processing target shelf image, privacy mask processing, etc.

[0118] Furthermore, the camera 3 includes a power supply unit 38. The power supply unit 38 is configured to include a secondary battery such as a NiCd battery, a NiMH battery, and a Li battery.

[0119] The power supply unit 38 includes a remaining battery capacity detection unit 39 that detects the remaining battery capacity of these secondary batteries.

[0120] The remaining capacity detection unit 39 manages the remaining battery capacity using a state of charge (SOC) or the like.

[0121] The power supply unit 38 may be configured to be able to receive power supplied from a commercial AC power source via an AC adapter or the like.

[0122] The image sensor IS is configured as, for example, a CCD type image sensor or a CMOS type image sensor.

[0123] The image sensor IS includes an imaging unit 41 , an image signal processing unit 42 , a sensor internal control unit 43 , a memory unit 45 , and a communication I / F 46 which can perform data communication with each other via a bus 47 .

[0124] The imaging unit 41 includes a pixel array unit in which pixels including photoelectric conversion elements such as photodiodes are two-dimensionally arranged, and a readout circuit that reads out an electrical signal obtained by photoelectric conversion from the pixels included in the pixel array unit and can output the electrical signal as a captured image signal.

[0125] The readout circuit performs, for example, correlated double sampling (CDS) processing and automatic gain control (AGC) processing on the electric signal obtained by the photoelectric conversion, and further performs analog / digital (A / D) conversion processing.

[0126] In the present embodiment, the imaging unit 41 captures a reference shelf image and a processing target shelf image. The processing target shelf image is obtained by imaging the imaging unit 41 every predetermined period of time (eg, every 5 minutes or every hour).

[0127] The imaging interval (predetermined time period) of the shelf image to be processed can be appropriately changed according to various factors. The details will be described later.

[0128] The image signal processing unit 42 performs pre-processing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, and the like on a captured image signal as digital data after A / D conversion processing.

[0129] In the preprocessing, a clamping process for clamping the R, G, and B black levels of the captured image signal to a predetermined level, a correction process between the R, G, and B color channels, and the like are performed. In the synchronization process, a color separation process for assigning all R, G, and B color components to the image data of each pixel is performed. For example, in the case of an imaging element using a Bayer array color filter, a demosaicing process is performed as the color separation process. In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data. In the resolution conversion process, a resolution conversion process is performed on the image data that has undergone various signal processing.

[0130] In the codec processing, for example, encoding processing for recording, communication, and file generation is performed on the image data that has undergone the above-mentioned various processing. In the codec processing, a moving image file in a format such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264 can be generated. It is also conceivable to generate a still image file in a format such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format). When the image sensor IS is a distance measurement sensor, the image signal processing unit 42 calculates distance information about the object based on two signals output from the image sensor IS as iToF (indirect time of flight), for example, and outputs a distance measurement image.

[0131] The sensor internal control unit 43 issues instructions to the imaging unit 41 and controls the execution of the imaging operation. Similarly, the sensor internal control unit 43 also controls the execution of the processing of the image signal processing unit 42.

[0132] The memory unit 45 can function as a so-called frame memory in which captured image data (raw (RAW) image data) obtained by the image signal processing unit 42 and image data after synchronization processing are stored.

[0133] The communication I / F 46 is an interface for communicating with the control unit 33, the memory unit 34, etc. located outside the image sensor IS. The communication I / F 46 performs communication for acquiring a program or the like to be executed by the image signal processing unit 42 from the outside and stores them in the memory unit 45 provided in the image sensor IS.

[0134] The image data and the like output from the image signal processing unit 42 are output from the communication I / F 46 of the image sensor IS.

[0135] <3. Hardware Configuration of Information Processing Device>

[0136] Will refer to Figure 4 The hardware configuration of information processing devices such as the cloud server 1 and the user terminal 2 of the information processing system 100 is described.

[0137] The information processing apparatus is equipped with a CPU 71. The CPU 71 functions as an arithmetic processing unit that performs the above-mentioned various processes, and performs various processes according to a program stored in a ROM 72 or a nonvolatile storage unit 74 such as an electrically erasable programmable read-only memory (EEP-ROM) or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also appropriately stores data necessary for the CPU 71 to perform various processes.

[0138] As described above, the CPU 71 as the information processing device of the cloud server 1 performs processing for cutting out a corresponding image from a reference shelf image, AI image processing for specifying the state of a target product in a corresponding image using an AI model, notification processing, and the like.

[0139] The storage unit such as the ROM and the RAM stores information about the AI ​​application and the AI ​​model used in the image processing performed using the AI ​​model by the CPU 71. The AI ​​application indicates an application for performing processing using the AI ​​model.

[0140] The CPU 71, the ROM 72, the RAM 73, and the nonvolatile storage unit 74 are connected to one another via a bus 83. An input / output interface (I / F) 75 is also connected to the bus 83.

[0141] An input unit 76 constituted by an operator or an operating device is connected to the input-output interface 75 .

[0142] For example, various operating elements or operating devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, and a remote controller are assumed as the input unit 76 .

[0143] The input unit 76 detects an operation by a user, and the CPU 71 analyzes a signal corresponding to the input operation.

[0144] A display unit 77 composed of a liquid crystal display (LCD), an organic electroluminescence (EL) panel, or the like, and a sound output unit 78 composed of a speaker or the like are connected to the input / output interface 75 integrally or individually.

[0145] The display unit 77 is a display unit that performs various displays, and is constituted by, for example, a display device provided in a casing of the computer device or a separate display device connected to the computer device.

[0146] The display unit 77 displays images for various image processing and moving images to be processed on the display screen based on instructions from the CPU 71. The display unit 77 also displays various operation menus, icons and messages based on instructions from the CPU 71, that is, a graphical user interface (GUI).

[0147] A storage unit 79 constituted by a hard disk or a solid-state memory or a communication unit 80 constituted by a modem may be connected to the input / output interface 75 .

[0148] The communication unit 80 performs communication processing via a transmission line such as the Internet, and communicates with various devices in a wired or wireless manner or via a bus.

[0149] A drive 81 is connected to the input / output interface 75 as necessary, or a removable storage medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately installed in the input / output interface 75 .

[0150] A data file such as a program for each process may be read from the removable storage medium 82 by the drive 81. The read data file is stored in the storage unit 79, or an image and sound included in the data file are output from the display unit 77 and the sound output unit 78. In addition, a computer program or the like read from the removable storage medium 82 may be installed in the storage unit 79 as necessary.

[0151] In the computer apparatus, for example, software for the processing in the present embodiment may be installed via network communication of the communication unit 80, or installed via the removable storage medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0152] Furthermore, images captured by the camera 3 and various processing results may be received via the storage unit 79 or the drive 81 and stored in the removable storage medium 82 .

[0153] The CPU 71 performs processing operations based on various programs, thereby executing information processing and communication processing required for the cloud server 1 and the user terminal 2 which are information processing devices equipped with the above-described arithmetic processing unit.

[0154] Each of the cloud server 1 and the user terminal 2 is not limited to Figure 4 The single computer device shown may be composed of a plurality of computer devices that are systemized. The plurality of computer devices may be systemized through a local area network (LAN) or the like, or may be set up in a remote location by using a virtual private network (VPN) using the Internet or the like. The plurality of computer devices may include a computer device that serves as a server group (cloud) that can be used by a cloud computing service.

[0155] <4. Functional Configuration>

[0156] The functional configurations of the camera 3 and the cloud server 1 will be described.

[0157] Figure 51 shows a functional configuration of the signal processing unit 36 ​​of the camera 3 . Some of the functions shown below can be realized by the control unit 33 of the camera 3 .

[0158] As shown in the figure, the signal processing unit 36 ​​functions as an image acquisition unit F1, an obstacle detection unit F2, an area specification unit F3, a cutout processing unit F4, a transmission processing unit F5, and a mask processing unit F6.

[0159] The image acquisition unit F1 acquires the reference shelf image and the processing target shelf image captured by the imaging unit 41 .

[0160] In addition, the image acquisition unit F1 optimizes the imaging interval by performing processing for changing the imaging interval of the processing target shelf image on the imaging unit 41 .

[0161] For example, when the remaining battery power detected by the remaining power detection unit 39 of the power supply unit 38 is lower than a predetermined threshold, the image acquisition unit F1 increases the imaging interval. In addition, when the remaining battery power recovers to a predetermined threshold or more, the image acquisition unit F1 decreases the imaging interval.

[0162] The obstacle detection unit F2 detects whether the product displayed in the display space of each shelf is appropriately shown in each shelf image acquired by the image acquisition unit F1, in other words, whether an obstacle such as a person is shown in front of the product.

[0163] When it is determined that an obstacle is displayed and the products in the display space are not properly displayed (occlusion has occurred), the shelf image is discarded and the imaging unit 41 is instructed to capture the shelf image again.

[0164] The area specifying unit F3 performs processing for specifying a difference area where there is a difference between the reference shelf image and the processing target shelf image. The difference area including the difference from the reference shelf image indicates that the display state of the product is different from when the store is open. Therefore, some work may be required for the product shown in the difference area.

[0165] For example, work includes tidying up, such as replenishing products or adjusting their orientation, or moving products to correct display position.

[0166] When a different area is detected, the area specifying unit F3 acquires information for specifying the position of the different area in the processing target shelf image. The information for specifying the position of the different area is, for example, information on the x-coordinate and the y-coordinate in the image.

[0167] For example, Figure 6As shown in FIG. 1 , a case where a rectangular different area is detected in a processing target shelf image whose number of pixels in the horizontal direction is X and whose number of pixels in the vertical direction is Y is described. In this case, the area specifying unit F3 acquires the coordinates (x1, y1) of the upper left pixel in the different area as the coordinates of the different area as information for specifying the position of the different area.

[0168] The area specifying unit F3 may also acquire the coordinates (x2, y2) of the lower right pixel in the different area as information for specifying the position of the different area, thereby making it possible to specify the size of the different area.

[0169] Another example of information for specifying the location of the difference area may be an identification (ID) assigned to each display space on a product shelf.

[0170] For example, Figure 7 As shown, a unique shelf ID (such as shelf T1, T2, ...) is assigned to each level of product shelves. Three types of products can be displayed on shelf T1, and a unique display space ID (such as display space Sp1, Sp2 or Sp3) is assigned to each display space.

[0171] Based on this premise, the area specifying unit F3 acquires the ID of the display space shown in the difference area where the difference has been detected. The display space ID can be specified from the coordinate information by using a correspondence table in which coordinates in the shelf image and the display space ID are associated with each other.

[0172] The cutout processing unit F4 cuts out a portion of the processing target shelf image as a partial image so as to include the difference area specified by the area specifying unit F3. The partial image is, for example, an image cut out as an area slightly larger than the difference area (see Figure 8 ).

[0173] Send processing unit F5 (see Figure 5 ) performs processing for sending the reference shelf image and the partial image to the cloud server 1. For example, it is assumed that sending of the reference shelf image is performed only for the reference shelf image captured before the store opens. Although it has been described that the first processing target shelf image is set as the reference shelf image when the second processing target shelf image is acquired, in this case, the reference shelf image as the first processing target shelf image is not sent to the cloud server 1.

[0174] That is, for example, the transmission processing unit F5 transmits the reference shelf image only once a day. However, in the case where the store is extensively renovated and the display position of the product changes significantly between the morning and the afternoon, the reference shelf image may be transmitted in both the morning and the afternoon.

[0175] As described above, the transmission processing unit F5 transmits the information for specifying the position of the difference area together with the partial image. In addition, instead of the information for specifying the position of the difference area, the coordinates of the partial image may be transmitted.

[0176] The transmission processing unit F5 also transmits compressed data on the reference shelf image and the partial image to be transmitted.

[0177] When an obstacle such as a person is shown in the shelf image or a partial image, the mask processing unit F6 performs privacy mask processing. For example, when a person is shown in the shelf image to hide a product, the shelf image is discarded and a new shelf image is captured. However, when a person is shown to the extent that the product is not hidden, the state of the product can be properly identified and there is no need to discard the shelf image. However, when the shelf image showing a person is sent to the cloud server 1, a privacy problem occurs. For this reason, the mask processing unit F6 performs privacy mask processing such as filling with a black image or mosaic processing on the area where the person is shown in the shelf image.

[0178] Next, we will refer to Fig. 9 The functional configuration of the cloud server 1 is described.

[0179] The CPU 71 of the cloud server 1 functions as a reception processing unit F11 , a corresponding image generation unit F12 , an image recognition processing unit F13 , a notification processing unit F14 , a count processing unit F15 , and an instruction processing unit F16 .

[0180] The receiving processing unit F11 performs processing for receiving a reference shelf image and a partial image as a part of a processing target shelf image. These images are received in a compressed state. The receiving processing unit F11 also receives position information about the partial image together with the partial image.

[0181] The corresponding image generation unit F12 performs processing for cutting out a region corresponding to the partial image in the reference shelf image as a corresponding image.

[0182] The generation of the corresponding image is a process for cutting out an image area as a comparison target from the reference shelf image in order to correctly identify the state of the product shown in the partial image of the processing target shelf image. Therefore, when the previously received partial image is regarded as (a part of) the reference shelf image to be compared, there is no need to perform the cutting out process by the corresponding image generation unit F12.

[0183] Specifically, for example, Fig.10As shown, a reference shelf image is captured at time t0 before the store opens, and the cloud server 1 receives the reference shelf image. Then, when the store opens at time t1, and the cloud server 1 receives a partial image (assumed to be partial image A) cut out from the processing target shelf image captured for the first time on that day at time t2 thereafter, there is a possibility that the state of the product Item1 shown in partial image A has changed. To confirm this, the cloud server 1 compares the corresponding image with the reference shelf image by cutting out the corresponding area from the reference shelf image ( Fig.10 The first comparison in ) is used to identify the status of product Item1.

[0184] Thereafter, at time t3, when the partial image (assumed to be partial image B) cut out from the processing target shelf image captured for the second time on that day is cut out from the same image area as the previous partial image, that is, when the coordinates of partial image B and the coordinates of partial image A are the same, the image to be compared with partial image B may be partial image A regarded as the reference shelf image. In this case, a comparison is made between the previous state (the state specified by partial image A) and the current state (the state specified by partial image B) for product Item1 ( Fig.10 2nd comparison in ).

[0185] Therefore, it is possible to prevent notification similar to that performed based on the result of the first comparison from being performed based on the result of the second comparison.

[0186] The reference shelf image captured at time t0 before the store opens is the optimal display state of the product and can be regarded as the target display state. In this case, even when a partial image B having the same coordinates as those of the partial image A is received as described above, the difference from the target display state of the product Item1 can be grasped and notified by comparing the corresponding image cut out from the reference shelf image captured at time t0 before the store opens with the partial image B.

[0187] In this case, appropriate notification can be issued each time based on the comparison result with the target display state. Fig.10 When the first comparison in the above example issues a notification but no work is performed to optimize the display state, a similar notification is issued again based on the second comparison. Therefore, notification can be performed for the case where the notification is ignored.

[0188] Image recognition processing unit F13 (see Fig. 9 ) Perform AI image processing for recognizing the state of a product by inputting the corresponding image and the partial image into the AI ​​model. The image recognition processing unit F13 obtains the recognition result of the state of the target product as the output of the AI ​​model.

[0189] There are several conceivable states of the target product.

[0190] First, in Fig.11 An example of a corresponding image of a reference shelf image is shown in . The corresponding image shows a state in which a plurality of breads having faces of giant pandas are displayed on a tray.

[0191] Next, an example of the status of the target product will be given. One is the status about the quantity of the target product. Fig.12 A partial image is shown, which shows a state in which all target products have been sold and the product quantity on the display is set to "0". This state is called an "out of stock state".

[0192] in addition, Fig.13 A partial image is shown, which shows a state in which some target products have been sold and the number of product items on display is less than a predetermined number (or a predetermined percentage). This state is called a "supply shortage state".

[0193] exist Fig.14 Another example of the state of the target product is shown in FIG. This figure shows a state related to the posture of the displayed target product, and shows a partial image that shows a state in which some target products are not properly oriented. This state is called an "inappropriate posture state".

[0194] exist Fig.15 . This figure shows a state regarding the position of the displayed target product, and shows a state in which a type of bread that should be displayed in a different display space and should not be inherently shown in the partial image is shown in the partial image. This state is called an "inappropriate display position state".

[0195] exist Fig.16 Another example of the state of the target product is shown in FIG. The figure shows a state in which the price tag on the target product that should be placed on the display is incorrect. Specifically, the price is "1200 yen" instead of "120 yen". This state is called an "inappropriate price tag state".

[0196] For the above-mentioned out-of-stock, short supply, inappropriate posture and inappropriate display position states, the cloud server 1 does not need to know what target product should be displayed in the partial image. That is, by inputting the corresponding image and the partial image into the AI ​​model, the out-of-stock state or the short supply state can be identified when the quantity of the product is reduced, the inappropriate posture state can be identified when the posture of the product is not uniform, and the inappropriate display position state can be identified when different types of products are mixed.

[0197] However, there is a concern that price tags may not be appropriate even when target products are arranged in an orderly manner, and the cloud server 1 needs to be able to acquire information on the correspondence relationship between products to be displayed in the display space and their prices.

[0198] Fig.17 The database shown is constructed in the storage unit 79 of the cloud server 1 or in the storage unit of another information processing device that the cloud server 1 can access.

[0199] As shown in the figure, the database stores data so that the correspondence between the camera ID for specifying the camera 3, the shelf ID, the display space ID, the product ID for specifying the product, and the price information can be identified.

[0200] Fig.17 An example of a case where there are multiple cameras 3 is shown. In addition, a single camera 3 may be configured to capture multiple shelf images by changing its viewing angle. In this case, an image ID (or viewing angle ID) is used instead of a camera ID.

[0201] The AI ​​model used by the image recognition processing unit F13 of the cloud server 1 to recognize the status of the product is trained in advance.

[0202] Here, the training of the AI ​​model will be described. An AI model can be prepared for each state of the product, or one AI model can be configured to recognize all states.

[0203] First, in order to grasp the "out of stock status" and "shortage status" of products, an AI model that has been trained using training data consisting of each pair of images showing one or more products and the number of products is used. In addition, the AI ​​model is trained using training data consisting of each pair of images showing only an empty pallet without showing a product and the number of products (zero). In addition, images of training data are prepared for each product to be recognized.

[0204] The AI ​​model obtained by training in this way is able to recognize the number of products shown in the input image. Therefore, the AI ​​model is able to recognize "out of stock state" and "supply shortage state". In identifying the supply shortage state, when the number of products on the display is less than the threshold, the threshold determined for each product can be used to determine the "supply shortage state", and training can be performed in advance using training data consisting of each pair of supply shortage state, image to be recognized, and "supply shortage state" label.

[0205] Next, the "inappropriate posture state" of the product is grasped using an AI model that has been trained using training data each consisting of pairs of images of the product captured from various angles and labels indicating whether the image shows the front.

[0206] The AI ​​model obtained in this way can identify the correct orientation of the product and can therefore identify the "inappropriate posture state" of the product.

[0207] Next, the "inappropriate display position status" of the product is grasped using an AI model that has been trained using training data including each pair consisting of an image of a product and its product name (or may be a product ID).

[0208] The AI ​​model obtained in this way can identify the differences between products, and therefore can identify the state where multiple products are displayed in a certain display space. Therefore, by identifying the state where multiple types of products are displayed in one display space, it is possible to identify an "inappropriate display position state".

[0209] Furthermore, by using a database storing the correspondence between display spaces and products that should be displayed in the display spaces, an “inappropriate display position state” can be appropriately identified even when only products other than products that should be inherently displayed are displayed in the display space.

[0210] Instead of using a database that stores the correspondence between display spaces and products that should be displayed in the display spaces, "inappropriate display position status" can be identified by performing optical character recognition (OCR) processing on price tags installed corresponding to the display spaces to detect mismatches between characters on the price tags and the identified product names.

[0211] Next, the "inappropriate price tag status" of the product is grasped using an AI model that has been trained using training data each consisting of pairs consisting of images of numbers and their numerical values.

[0212] The AI ​​model obtained in this way can recognize the price written on the price tag. Therefore, it can be determined whether the price that should be given to the product matches the price read from the image, and identify the "inappropriate price tag status".

[0213] Notification processing unit F14 (see Fig. 9 ) performs notification processing corresponding to each product and each product state. For example, in the case of an out-of-stock state or a supply shortage state, a notification processing for prompting a replenishment work to add a product to the display space is performed. In addition, in the case of an inappropriate posture state or an inappropriate display position state, a notification processing for prompting a work to display the product again or a rearrangement work is performed. In addition, in the case of an inappropriate price tag state, a notification processing for prompting a work to re-paste or rewrite the price tag is performed.

[0214] The notification processing of the notification processing unit F14 may include an alarm processing.

[0215] The notification process performed by the notification processing unit F14 is performed on the user terminal 2. Therefore, the administrator using the user terminal 2 can perform a predetermined work by himself or by giving instructions to employees to standardize the display status of the product.

[0216] The count processing unit F15 counts the number of times the notification process is performed for each product and each product state. The count result is stored in the storage unit 79 of the cloud server 1 or in the storage unit of another information processing device.

[0217] exist Fig.18 An example of the counting result is shown in FIG.

[0218] like Figure 7 As shown, shelf T1 has three display spaces, and different products are displayed in the display spaces. The three products are product Item1, product Item2, and product Item3.

[0219] The shelf T2 also has three display spaces, and products Item4, Item5, and Item6 are displayed in the display spaces, respectively.

[0220] Fig.18 The example shown is an example in which the processing target shelf image is captured four times during the business hours of the store. Therefore, the maximum count number of times for each notification is 4. For the product Item5 displayed on the shelf T2, the notification of the out-of-stock state is issued four times. That is, it is indicated that the product Item5 is a popular product and is likely to be sold quickly even when it is displayed.

[0221] The count information about such notifications is important information that can be used for product sales strategies through analysis.

[0222] For product Item5, out-of-stock notifications have been issued in all four notification opportunities, and thus four opportunities to capture processing target shelf images during business hours are likely to be too few in terms of monitoring sales of product Item5.

[0223] The instruction processing unit F16 executes a process for instructing the camera 3 on the imaging interval for processing the target shelf image according to the counting result of the counting processing unit F15. Fig.18 In the example shown, the instruction processing unit F16 determines that the product Item5 needs to be monitored more frequently, and issues an instruction to increase the number of opportunities to capture processing target shelf images during business hours to eight times.

[0224] In addition, other situations where the instruction processing unit F16 instructs the imaging interval of processing the target shelf image may also be considered. For example, when the battery information of the camera 3 is managed in the cloud server 1, the instruction processing unit F16 may instruct the camera 3 to change the imaging interval based on the remaining battery power.

[0225] <5. Processing Flow>

[0226] Various processes performed by the camera 3 and the cloud server 1 will be described with reference to the drawings. In the following description, the processes performed by the signal processing unit 36 ​​of the camera 3 may be performed by the control unit 33 of the camera 3 .

[0227] <5-1. Processing Related to Acquisition of Reference Shelf Image>

[0228] first, Fig.19 An example of processing performed by the camera 3 before the store opens is shown.

[0229] In step S101 , the control unit 33 of the camera 3 executes various processes for starting the camera 3 .

[0230] Next, in step S102 , the control unit 33 of the camera 3 performs imaging control for capturing a reference shelf image. The reference shelf image obtained by the imaging operation of the imaging unit 41 is stored in the memory unit 34 .

[0231] In step S103 , the signal processing unit 36 ​​of the camera 3 reads out the reference shelf image from the memory unit 34 , and in step S104 , the obstacle detection unit F2 performs an obstacle detection process to determine whether an obstacle is detected.

[0232] When determining that an obstacle has been detected, the signal processing unit 36 ​​of the camera 3 deletes the reference shelf image stored in the memory unit 34 in step S105 , and waits until a predetermined period of time has elapsed in step S106 .

[0233] When it is determined in step S106 that the predetermined period of time has elapsed, the signal processing unit 36 ​​returns to the process of step S102 and captures the reference shelf image again.

[0234] On the other hand, when it is determined in step S104 that an obstacle is not detected, the signal processing unit 36 ​​of the camera 3 performs compression processing on the reference shelf image in step S107 .

[0235] Next, in step S108, the signal processing unit 36 ​​of the camera 3 performs a process for transmitting the compressed reference shelf image to the cloud server 1. When the camera 3 is capable of switching between a mode for outputting an image and a mode for outputting only metadata, the process for switching to the mode for outputting an image may be performed before the transmission process in step S108.

[0236] In step S109, the signal processing unit 36 ​​of the camera 3 switches to the sleep mode. Thus, the camera 3 can reduce power consumption until the next shelf image is captured.

[0237] By executing Fig.19 With the process shown, the camera 3 may send a reference shelf image in which the target display state is imaged to the cloud server 1 .

[0238] When an obstacle has been detected in the reference shelf image but is detected at a position that does not interfere with image recognition of the product, it is determined in the process of step S104 that no obstacle is detected. When the detected obstacle is a person, a privacy mask process may be performed before the process of step S107.

[0239] <5-2. Processing Related to Acquisition of Target Shelf Image>

[0240] Next, we will refer to Fig. 20 An example of a process performed by the camera 3 to capture a process target shelf image during business hours of a store is described.

[0241] In step S201, the control unit 33 of the camera 3 determines whether a predetermined period of time has passed since the previous capture of the shelf image. When it is determined that the predetermined period of time has not passed, the control unit 33 of the camera 3 returns to the process of step S201.

[0242] On the other hand, when it is determined that the predetermined period of time (for example, two hours) has passed, the control unit 33 of the camera 3 performs imaging control for capturing the processing target shelf image in step S202. The reference shelf image obtained by the imaging operation of the imaging unit 41 is stored in the memory unit 34.

[0243] In step S203, the signal processing unit 36 ​​of the camera 3 reads out the processing target shelf image from the memory unit 34, and in step S204, the signal processing unit 36 ​​performs obstacle detection processing through the obstacle detection unit F2 to determine whether an obstacle is detected in the processing target shelf image.

[0244] When determining that an obstacle has been detected, the signal processing unit 36 ​​of the camera 3 deletes the processing target shelf image stored in the memory unit 34 in step S205 , and waits until a predetermined period of time has elapsed in step S206 .

[0245] When it is determined in step S206 that the predetermined period of time has elapsed, the signal processing unit 36 ​​returns to the process of step S202 and captures the processing target shelf image again.

[0246] On the other hand, when it is determined that no obstacle is detected, the signal processing unit 36 ​​of the camera 3 reads out a previous shelf image from the memory unit 34 as a reference shelf image in step S207, and performs processing as the area designation unit F3 in step S208 for comparing the reference shelf image with the current processing target shelf image to detect a difference.

[0247] In step S209, the signal processing unit 36 ​​of the camera 3 determines whether a difference area has been detected. When determining that a difference area has not been detected, the signal processing unit 36 ​​proceeds to the process of step S109 and shifts to the sleep mode.

[0248] On the other hand, when it is determined that a different area has been detected, the signal processing unit 36 ​​of the camera 3 cuts out a partial area of ​​the processing target shelf image as a partial image in step S210 so that the different area is included in step S210, and compresses the partial image in step S211.

[0249] In step S212 , the signal processing unit 36 ​​of the camera 3 performs processing for transmitting the compressed partial image to the cloud server 1 .

[0250] In step S213 , the signal processing unit 36 ​​of the camera 3 deletes the current reference shelf image from the memory unit 34 , and stores the current processing target shelf image in the memory unit 34 as the next reference shelf image.

[0251] In step S109 , after the signal processing unit 36 ​​of the camera 3 shifts to the sleep mode, the signal processing unit 36 ​​returns to the process of step S201 .

[0252] Camera 3 Execution Fig. 20 The processing shown, and therefore the image area that has changed in the processing target shelf image can be sent to the cloud server 1 as a partial image.

[0253] <5-3. Processing of Notification>

[0254] Next, Fig.21 and 22An example of the processing performed by the cloud server 1 to issue various notifications is shown in . The processing described below is the processing performed by the cloud server 1 on one camera 3. When multiple cameras 3 send images to the cloud server 1, the cloud server 1 performs the following processing on each camera 3.

[0255] First, in Fig.21 In step S301 , the CPU 71 of the cloud server 1 receives a reference shelf image captured before the store opens from the camera 3 and stores it in the storage unit 79 .

[0256] Next, in step S302, the CPU 71 of the cloud server 1 waits until a partial image is received from the camera 3. When determining that the partial image has been received from the camera 3, the CPU 71 of the cloud server 1 stores the received partial image in the storage unit 79 in step S303. The cloud server 1 receives the position information about the partial image from the camera 3 together with the partial image.

[0257] Here, the reference shelf image and the partial image stored in the storage unit 79 are compressed image data. Therefore, the CPU 71 of the cloud server 1 reads out the partial image from the storage unit 79 in step S304, and performs decompression processing on the partial image in the subsequent step S305.

[0258] In step S306 , the CPU 71 of the cloud server 1 reads out the reference shelf image from the storage unit 79 , and performs decompression processing.

[0259] In step S307 , the CPU 71 of the cloud server 1 cuts out a corresponding image from the reference shelf image based on the position information of the partial image.

[0260] In step S308, the CPU 71 of the cloud server 1 inputs the corresponding image and the partial image into the AI ​​model to perform image recognition processing on the state of the product shown in the partial image. As a result of the image recognition processing, the state of the product is output from the AI ​​model. In addition to the above-mentioned "out of stock state", "supply shortage state", "inappropriate posture state", "inappropriate display position state" and "inappropriate price tag state", the state of the product includes a "normal state" output when there is no problem. The CPU 71 of the cloud server 1 performs the following processing based on the recognition result of the state of the product output from the AI ​​model.

[0261] Specifically, in Fig. 22 In step S309 , the CPU 71 of the cloud server 1 determines whether the recognition result of the product status output from the AI ​​model indicates an out-of-stock status or a supply shortage status.

[0262] When it is determined that the identification result of the product status indicates an out-of-stock status or a supply shortage status, the CPU 71 of the cloud server 1 performs a notification process about product replenishment to the user terminal 2 in step S310. At this time, the notification method may be different for the out-of-stock status and the supply shortage status. For example, the notification of the out-of-stock status may be issued as an alarm notification process and may be displayed on the user terminal 2 in a more prominent manner than the supply shortage status.

[0263] When the product is sold, the status will eventually transition to the short supply status, but the timing may be different for each product. For example, for a product that requires time-consuming processing (such as baking bread) to replenish the product, it may be determined early that the product status is the short supply status. Additionally, for a product that can be simply replenished by bringing the product from the backyard and displaying it on the product shelf, it may be determined later that the product status is the short supply status.

[0264] After executing the notification process, the CPU 71 of the cloud server 1 stores the notification history in step S311. As described above, this process is required to adjust the imaging interval according to the number of notifications. The same applies to the following process for storing the notification history.

[0265] After executing the history storage process, or after determining in step S309 that the product status is not the out-of-stock status or the short-supply status, the CPU 71 of the cloud server 1 determines in step S312 whether the recognition result of the product status indicates an inappropriate posture state.

[0266] When determining that the recognition result of the product state indicates an inappropriate posture state, the CPU 71 of the cloud server 1 executes a notification process for prompting arrangement of the product in step S313 , and executes a process for storing a history of the notification process in step S314 .

[0267] After executing the history storage process, or after determining in step S312 that the product state is not an inappropriate posture state, the CPU 71 of the cloud server 1 determines in step S315 whether the recognition result of the product state indicates an inappropriate display position state.

[0268] When determining that the recognition result of the product status indicates an inappropriate display position status, the CPU 71 of the cloud server 1 executes a notification process for moving the product in step S316 , and executes a process for storing a history of the notification process in step S317 .

[0269] After executing the history storage process, or after determining in step S315 that the product status is not an inappropriate display position status, the CPU 71 of the cloud server 1 determines in step S318 whether the identification result of the product status indicates an inappropriate price tag status.

[0270] When determining that the recognition result of the product status indicates an inappropriate price tag status, the CPU 71 of the cloud server 1 performs a notification process for correcting the price tag in step S319 , and performs a process for storing a history of the notification process in step S320 .

[0271] After executing the history storage process, or after determining in step S318 that the product status is not the inappropriate price tag status, the CPU 71 of the cloud server 1 returns to step S302 and waits until a predetermined period of time has passed and a partial image is sent from the camera 3 again.

[0272] In this way, a partial image sent from the camera 3 is received at predetermined time intervals, recognition processing for recognizing the status of a product is performed, and appropriate notification processing is performed based on the result. Thus, the administrator of the products in the store can appropriately maintain the display status of the products. In addition, the administrator can view the notification content and perform appropriate work without performing the work of confirming the display status of the product, and thus a reduction in workload and an improvement in work efficiency can be achieved.

[0273] <5-4. Processing for Optimizing Imaging Intervals According to the Number of Notifications>

[0274] Next, Fig.23 An example of processing performed by the cloud server 1 to optimize the imaging interval according to the number of notifications of the status of the product is shown.

[0275] In step S401 , the CPU 71 of the cloud server 1 acquires the current time.

[0276] Next, in step S402, the CPU 71 of the cloud server 1 determines whether the current time exceeds the store closing time. When the current time does not exceed the store closing time, that is, when it is determined that the store is open, the CPU 71 of the cloud server 1 returns to the process of step S401.

[0277] On the other hand, when it is determined that the current time exceeds the store closing time, the following processing is performed as the processing after the store closes. Specifically, in step S403, the CPU 71 of the cloud server 1 determines whether the notification indicating the same product and content has occurred equal to or more than a predetermined number of times in a day. For example, the notification includes information about Fig.18 The out-of-stock status notification of the product Item5 is shown.

[0278] When determining that the same type of notification has occurred equal to or greater than a predetermined number of times, the CPU 71 of the cloud server 1 generates a change instruction for reducing the imaging interval of the camera 3 that images the relevant product in step S404 and sends the change instruction to the camera 3.

[0279] <5-5. Processing for Optimizing Imaging Interval>

[0280] at last, Fig.24 An example of processing performed by the camera 3 to optimize the imaging interval is shown in FIG.

[0281] In step S501 , the signal processing unit 36 ​​of the camera 3 determines whether an instruction to change the imaging interval has been received from the cloud server 1 .

[0282] When determining that the change instruction is received from the cloud server 1 , the signal processing unit 36 ​​of the camera 3 performs processing for changing the imaging interval according to the instruction in step S502 , and returns to the processing of step S501 .

[0283] On the other hand, when determining that the change instruction has not been received, the signal processing unit 36 ​​of the camera 3 acquires information on the remaining battery level in step S503 , and determines whether the remaining battery level is less than a predetermined threshold in the subsequent step S504 .

[0284] When it is determined that the remaining battery power is less than the predetermined value, the signal processing unit 36 ​​of the camera 3 performs a change process for increasing the imaging interval in step S502, and returns to the process of step S501. When the imaging interval has been reduced according to the determination process of step S501, the process of step S502 may be avoided without performing step S502, or the imaging interval may be further reduced in the process of step S502.

[0285] When it is determined in step S504 that the remaining battery charge is equal to or greater than the predetermined value, the signal processing unit 36 ​​of the camera 3 determines in step S505 whether a change to increase the imaging interval has been performed. When it is determined that the change has been performed, that is, when it is determined that the processing for increasing the imaging interval has been performed due to the low remaining battery charge, the remaining battery charge has returned to being equal to or greater than the predetermined threshold value, and therefore, the signal processing unit 36 ​​of the camera 3 performs a processing for returning (reducing) the imaging interval to the original imaging interval in step S506, and returns to the processing of step S501.

[0286] The imaging interval is optimized in the camera 3 in this way, and thus the state of the product can be appropriately acquired, and a state in which imaging cannot be performed by the camera 3 due to battery exhaustion is avoided.

[0287] <6. Second Embodiment>

[0288] In the second embodiment, unlike the first embodiment, processing for comparing the reference shelf image and the processing target shelf image, processing for cutting out a partial image, processing for specifying position information about a partial image in the processing target shelf image, and the like are performed in the image sensor IS instead of in the signal processing unit 36.

[0289] Hereinafter, differences from the first embodiment will be mainly described, and descriptions of portions similar to those in the first embodiment will be appropriately omitted.

[0290] Fig.25 A block diagram showing an example of the internal configuration of the camera 3A is shown in FIG.

[0291] The camera 3A includes an imaging optical system 31 , an optical system driving unit 32 , an image sensor IS, a control unit 33 , a memory unit 34 , a communication unit 35 , a signal processing unit 36 ​​, and a power supply unit 38 .

[0292] The image sensor IS, the control unit 33 , the memory unit 34 , the communication unit 35 , and the remaining charge detection unit 39 of the power supply unit 38 are connected to one another via a bus 37 , and can perform data communication with one another.

[0293] The configurations of the imaging optical system 31 , the optical system driving unit 32 , the control unit 33 , the memory unit 34 , the communication unit 35 , and the power supply unit 38 are the same as those in the first embodiment, and thus descriptions thereof are omitted.

[0294] The signal processing unit 36 ​​performs a privacy mask process, an image compression process, and a transmission process. A repeated description of the process will be omitted.

[0295] The image sensor IS is configured as, for example, a CCD type image sensor or a CMOS type image sensor.

[0296] Furthermore, in the image sensor IS in the present embodiment, various AI image processing such as object detection processing and image recognition processing using an AI model is performed.

[0297] The image sensor IS includes an imaging unit 41, an image signal processing unit 42, a sensor internal control unit 43, an AI image processing unit 44, a memory unit 45, and a communication I / F 46. These units can perform data communication with each other via a bus 47.

[0298] The imaging unit 41 , the image signal processing unit 42 , and the sensor internal control unit 43 are configured similarly to those in the first embodiment, and thus description thereof will be omitted.

[0299] The AI ​​image processing unit 44 performs image recognition processing as AI image processing on the captured image.

[0300] In this embodiment, the AI ​​image processing unit 44 is implemented by a digital signal processor (DSP).

[0301] As AI image processing, the AI ​​image processing unit 44 performs processing for comparing a reference shelf image with a processing target shelf image to detect a different area, processing for cutting out a partial image, processing for specifying position information of a partial image in a processing target shelf image, and the like.

[0302] In order to realize these various types of AI image processing by the AI ​​image processing unit 44, the camera 3A is configured to be able to change the algorithm of the AI ​​image processing performed by the AI ​​image processing unit 44. In other words, the function of the AI ​​image processing can be switched by switching the AI ​​model used in the AI ​​image processing.

[0303] For example, the AI ​​model is switched in response to an instruction from the control unit 33 of the camera 3A. In addition, when the AI ​​model is switched, the AI ​​model can be switched by selecting one from a plurality of AI models stored in the memory unit 45 or by receiving and deploying the AI ​​model from a server device such as the cloud server 1. By receiving the AI ​​model from the cloud server 1 or the like each time the AI ​​model is switched, the capacity of the memory unit 45 can be reduced, which leads to miniaturization, power saving, and cost reduction. In addition, when a plurality of AI models are stored in the memory unit 45, an increase in the amount of communication associated with the transmission and reception of the AI ​​model can be suppressed.

[0304] The memory unit 45 may be used to temporarily store data used by the AI ​​image processing unit 44 during the AI ​​image processing process.

[0305] The memory unit 45 also stores information about the AI ​​application and the AI ​​model used by the AI ​​image processing unit 44 .

[0306] Information about the AI ​​application and the AI ​​model may be deployed in the memory unit 45 as a container or the like by using the container technology to be described below, or may be deployed using the microservice technology. By deploying the AI ​​model for AI image processing in the memory unit 45, it is possible to change the function type of the AI ​​image processing or change the AI ​​model to an AI model whose performance has been improved by retraining.

[0307] As described above, in the present embodiment, description is made based on the example of an AI model and an AI application for image recognition processing, but it is not limited thereto and can also be applied to a program executed using AI technology.

[0308] In addition, when the capacity of the memory unit 45 is small, information about the AI ​​application and the AI ​​model can be deployed as a container, etc. in the memory unit 34 outside the image sensor IS by using container technology, and then only the AI ​​model can be stored in the memory unit 45 set in the image sensor IS via the communication I / F 46.

[0309] The communication I / F 46 is an interface for communicating with the control unit 33, the memory unit 34, and the like provided outside the image sensor IS. The communication I / F 46 performs communication for acquiring the program executed by the image signal processing unit 42, the AI ​​application program and the AI ​​model used by the AI ​​image processing unit 44, and the like from the outside, and stores them in the memory unit 45 provided in the image sensor IS.

[0310] Thereby, the AI ​​model is stored in the memory unit 45 provided in the image sensor IS, and can be used by the AI ​​image processing unit 44 .

[0311] The AI ​​image processing unit 44 performs predetermined image recognition processing using the AI ​​application or AI model obtained in this way to recognize the object according to the purpose. Specifically, the image recognition processing is performed on a person as the object to determine whether an obstacle such as a person is captured.

[0312] The recognition result information of the AI ​​image processing in the AI ​​image processing unit 44 is output to the outside of the image sensor IS via the communication I / F 46 .

[0313] That is, the communication I / F 46 of the image sensor IS outputs not only the image data output from the image signal processing unit 42 but also the recognition result information on the AI ​​image processing.

[0314] The communication I / F 46 of the image sensor IS may output only the image data or the recognition result information.

[0315] For example, when the retraining function of the AI ​​model is used, captured image data used in the retraining function is uploaded from the image sensor IS to the cloud server 1 or the like via the communication I / F 46 and the communication unit 35 .

[0316] In addition, when inference using the AI ​​model is performed, recognition result information about the AI ​​image processing is output from the image sensor IS to other information processing devices outside the camera 3A via the communication I / F 46 and the communication unit 35 .

[0317] Fig.26 The functional configuration of the AI ​​image processing unit 44 is shown in FIG.

[0318] The AI ​​image processing unit 44 functions as an image acquisition unit F1 , an obstacle detection unit F2 , an area specification unit F3 , and a cutout processing unit F4 . The transmission processing unit F5 and the mask processing unit F6 are functions of the signal processing unit 36 ​​.

[0319] A repeated description of each function will be omitted.

[0320] Will refer to Figures 19 to 24 The drawings of describe the processing performed by the camera 3A in the present embodiment. The differences from the first embodiment will be mainly described.

[0321] Fig.19 The illustrated processing is an example of processing performed by the camera 3A before the store opens as described above.

[0322] The reference shelf image obtained by the imaging control in step S102 is stored in the memory unit 45 in the image sensor IS.

[0323] The reading of the reference shelf image in step S103 , the obstacle detection process in step S104 , the reference shelf image deletion process in step S105 , and the standby process in step S106 are performed by the AI ​​image processing unit 44 instead of the signal processing unit 36 ​​.

[0324] As in the first embodiment, the compression process in step S107 and the transmission process in step S108 are performed by the signal processing unit 36 ​​.

[0325] As mentioned above, Fig. 20 The illustrated processing is an example of processing performed by the camera 3A during the business hours of the store.

[0326] The processing target shelf image obtained by the imaging control in step S202 is stored in the memory unit 45 in the image sensor IS.

[0327] Reading of the processing target shelf image in step S203 , obstacle detection processing in step S204 , processing target shelf image deletion processing in step S205 , and standby processing in step S206 are performed by the AI ​​image processing unit 44 instead of the signal processing unit 36 ​​.

[0328] Furthermore, the reading of the reference shelf image in step S207, the different area detection processing in step S208, the determination processing in step S209, the processing for cutting out a partial image in step S210, and the storage processing in step S213 are also performed by the AI ​​image processing unit 44 instead of the signal processing unit 36. In the storage processing in step S213, the processing target shelf image is stored in the memory unit 45 as the next reference shelf image.

[0329] Fig.21 , Fig. 22 , Fig.23 and Fig.24 The processing shown in is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0330] As in the present embodiment, various processes in the camera 3A can be performed by the AI ​​image processing unit 44 in the image sensor IS. Specifically, since the shelf image determined to be discarded by the obstacle detection processing includes a person and is discarded, it does not have to be output to the outside of the image sensor IS, and thus a certain degree of privacy protection can be achieved.

[0331] <7. Third Embodiment>

[0332] In the third embodiment, unlike the first and second embodiments, some of the processes performed by the cloud server 1 are performed by the camera 3B.

[0333] Fig. 27 is a block diagram showing an example of the internal configuration of the camera 3B.

[0334] The camera 3B includes an imaging optical system 31, an optical system driving unit 32, an image sensor IS, a control unit 33, a memory unit 34, a communication unit 35, and a power supply unit 38. That is, the camera 3B does not include the signal processing unit 36.

[0335] Similar to the second embodiment, the image sensor IS is configured to include an imaging unit 41, an image signal processing unit 42, a sensor internal control unit 43, an AI image processing unit 44, a memory unit 45, and a communication I / F 46. These units can perform data communication with each other via a bus 47.

[0336] The AI ​​image processing unit 44 performs image recognition processing as AI image processing on the captured image.

[0337] As AI image processing, the AI ​​image processing unit 44 performs processing for comparing a reference shelf image with a processing target shelf image to detect a difference area, processing for cutting out a partial image, processing for specifying position information of a partial image in the processing target shelf image, processing for cutting out a corresponding image from the reference shelf image, processing for performing image recognition processing using the corresponding image and the partial image as input images and outputting a recognition result about the status of a product, and processing for determining notification content according to the status of the recognized product.

[0338] In order to realize these various types of AI image processing by the AI ​​image processing unit 44, the camera 3B is configured to be able to change the algorithm of the AI ​​image processing performed by the AI ​​image processing unit 44. The algorithm of the AI ​​image processing is changed in response to an instruction from the control unit 33 of the camera 3B, for example.

[0339] In addition, each AI model and AI application may be deployed in the memory unit 45 using the container technology to be described below, or may be deployed using the microservice technology.

[0340] Fig.28 The functional configuration of the AI ​​image processing unit 44 is shown in FIG.

[0341] As shown in the figure, the AI ​​image processing unit 44 functions as an image acquisition unit F1, an obstacle detection unit F2, an area specification unit F3, a cutout processing unit F4, a corresponding image generation unit F12, an image recognition processing unit F13 and a transmission processing unit F5.

[0342] That is, the camera 3B captures an image of a product displayed on a product shelf, detects a change in the display state of the product based on the image, and determines whether notification is necessary. When notification is necessary, the camera 3B functions as a transmission processing unit F5 that transmits notification content to the cloud server 1.

[0343] When performing the privacy mask processing, the AI ​​image processing unit 44 can function as the mask processing unit F6.

[0344] When optimizing the imaging interval, the AI ​​image processing unit 44 can function as the count processing unit F15 or the indication processing unit F16.

[0345] Fig.29 2 shows the functional configuration of the cloud server 1 in this embodiment.

[0346] The CPU 71 of the cloud server 1 functions as a notification processing unit F14 .

[0347] That is, the cloud server 1 implements only the notification processing function among the above functions.

[0348] The processing performed by the camera 3B and the cloud server 1 will be described with reference to the accompanying drawings. For the same processing as the first embodiment, the same step number is used, and its description will be omitted appropriately. In addition, in each figure, the same step number is used for the processing performed by different processing entities as long as the processing content is the same.

[0349] first, Fig.30 , a process for acquiring reference shelf images performed before a store opens is shown.

[0350] In step S101 , the control unit 33 of the camera 3B executes a startup process of the camera 3B.

[0351] Next, in step S102 , the control unit 33 of the camera 3B performs imaging control for capturing a reference shelf image. The reference shelf image obtained by the imaging operation of the imaging unit 41 is stored in the memory unit 45 .

[0352] The AI ​​image processing unit 44 of the camera 3B reads out the reference shelf image from the memory unit 45 in step S103 , and performs obstacle detection processing by the obstacle detection unit F2 in step S104 to determine whether an obstacle is detected.

[0353] When determining that an obstacle has been detected, the AI ​​image processing unit 44 of the camera 3B deletes the reference shelf image stored in the memory unit 45 in step S105 , waits until a predetermined period of time has passed in step S106 , and then returns to the process of step S102 .

[0354] On the other hand, when it is determined in step S104 that no obstacle is detected, the A1 image processing unit 44 of the camera 3B shifts to the sleep mode in step S109.

[0355] That is, in the camera 3B, the compression process in step S107 and the transmission process in step S108 are omitted.

[0356] Compression processing may be performed, and in this case, the data capacity of the shelf image stored in the memory unit 45 can be reduced and the memory unit 45 can be effectively used.

[0357] Next, we will refer to Fig.31 and Fig.32 An example of processing performed by the camera 3B to capture a processing target shelf image during business hours of a store and detect the status of a product to determine notification content is described.

[0358] In step S201, the control unit 33 of the camera 3B determines whether a predetermined time period has passed since the previously captured shelf image. When the predetermined time period has not passed, the control unit 33 of the camera 3B returns to the process of step S201.

[0359] When it is determined that the predetermined period of time has elapsed, the control unit 33 of the camera 3B performs imaging control for capturing a processing target shelf image in step S202 . The reference shelf image obtained by the imaging operation of the imaging unit 41 is stored in the memory unit 45 .

[0360] In step S203, the AI ​​image processing unit 44 of the camera 3B reads out the processing target shelf image from the memory unit 45, and in step S204, the AI ​​image processing unit 44 performs obstacle detection processing through the obstacle detection unit F2 to determine whether an obstacle is detected in the processing target shelf image.

[0361] When determining that an obstacle has been detected, the A1 image processing unit 44 of the camera 3B deletes the processing target shelf image stored in the memory unit 45 in step S205, waits until a predetermined period of time has passed in step S206, and then returns to the process of step S201.

[0362] On the other hand, when it is determined that no obstacle is detected, the AI ​​image processing unit 44 of the camera 3B reads out a previous shelf image from the memory unit 45 as a reference shelf image in step S207, performs processing as an area specifying unit F3 in step S208 for comparing the reference shelf image with the current processing target shelf image to detect a difference therebetween, and determines whether a difference area is detected in step S209.

[0363] When determining that no difference area is detected, the AI ​​image processing unit 44 returns to the process of step S201. At this time, the control unit 33 of the camera 3B may cause the camera 3B to shift to the sleep mode.

[0364] On the other hand, when it is determined that a difference area has been detected, the AI ​​image processing unit 44 of the camera 3B cuts out a portion of the processing target shelf image as a partial image in step S210 so as to include the difference area, deletes the current reference shelf image from the memory unit 45 in step S213, and stores the current processing target shelf image in the memory unit 45 as the next reference shelf image.

[0365] That is, the camera 3B does not perform the compression process of the partial image in step S211 and the transmission process in step S212.

[0366] Next, the camera 3B performs the processing performed by the cloud server 1 in the first embodiment. Specifically, the AI ​​image processing unit 44 of the camera 3B performs Fig.32 The reference shelf image is read out from the memory unit 45 in step S601, and the corresponding image is cut out from the reference shelf image based on the position information about the partial image in step S307.

[0367] In step S308, the AI ​​image processing unit 44 of the camera 3B inputs the corresponding image and the partial image to the AI ​​model to perform image recognition processing on the state of the product shown in the partial image. As a result of the image recognition processing, the state of the product is output from the AI ​​model. Similar to other embodiments, the state of the product includes "out of stock state", "supply shortage state", "inappropriate posture state", "inappropriate display position state", "inappropriate price tag state" and "normal state".

[0368] In step S309 , the AI ​​image processing unit 44 of the camera 3B determines whether the recognition result of the product status output from the AI ​​model indicates an out-of-stock status or a supply shortage status.

[0369] When it is determined that the recognition result of the product status indicates an out-of-stock status or a supply shortage status, the AI ​​image processing unit 44 of the camera 3B transmits a notification request for product replenishment to the cloud server 1 in step S602. At this time, regarding the notification request transmitted to the cloud server 1, the out-of-stock status or the supply shortage status can be distinguished from each other.

[0370] After sending the notification request, the AI ​​image processing unit 44 of the camera 3B stores the notification history in step S311 .

[0371] After executing the history storage process, or after determining in step S309 that the product status is not the out-of-stock status or the short-supply status, the AI ​​image processing unit 44 of the camera 3B determines in step S312 whether the recognition result of the product status indicates an inappropriate posture state.

[0372] When determining that the recognition result of the product state indicates an inappropriate posture state, the AI ​​image processing unit 44 of the camera 3B transmits a notification request of product arrangement to the cloud server 1 in step S603 , and performs processing for storing a notification history in step S314 .

[0373] After executing the history storage process, or after determining in step S312 that the product state is not an inappropriate posture state, the AI ​​image processing unit 44 of the camera 3B determines in step S315 whether the recognition result of the product state indicates an inappropriate display position state.

[0374] When determining that the recognition result of the product status indicates an inappropriate display position status, the AI ​​image processing unit 44 of the camera 3B transmits a notification request of product movement to the cloud server 1 in step S604 , and performs processing for storing a notification history in step S317 .

[0375] After executing the history storage process, or after determining in step S315 that the product state is not an inappropriate display position state, the AI ​​image processing unit 44 of the camera 3B determines in step S318 whether the recognition result of the product state indicates an inappropriate price tag state.

[0376] When it is determined that the recognition result of the product status indicates an inappropriate price tag status, the AI ​​image processing unit 44 of the camera 3B sends a notification request for price tag correction to the cloud server 1 in step S605, and performs processing for storing a notification history in step S320.

[0377] After executing the history storage process, or after determining in step S318 that the product state is not the inappropriate price tag state, the AI ​​image processing unit 44 of the camera 3B returns to the process of step S201 and waits until a predetermined period of time has passed.

[0378] Next, we will refer to Fig.33 The processing performed by the cloud server 1 is described.

[0379] In step S701 , the CPU 71 of the cloud server 1 determines whether a notification request has been received from the camera 3B.

[0380] When determining that the notification request has not been received, the CPU 71 of the cloud server 1 repeats the process of step S701 .

[0381] On the other hand, when determining that the notification request has been received, the CPU 71 of the cloud server 1 performs notification processing on the user terminal 2 according to the notification request in step S702 .

[0382] In this way, in the third embodiment, the camera 3B performs every process except the notification process of the user terminal 2. Then, the cloud server 1 performs the notification process only according to the request of the camera 3B.

[0383] Thereby, the captured shelf image is not output to the outside of the image sensor IS, and privacy protection can be strongly achieved.

[0384] In addition, many processes are performed in the camera 3B, and therefore the processing burden on the cloud server 1 where processes tend to be concentrated can be reduced, and the amount of communication can also be reduced.

[0385] <8. Fourth embodiment>

[0386] Unlike the above-described embodiments, the information processing system 100 according to the fourth embodiment includes a fog server 4 (see Fig.34 ).

[0387] For example, it is assumed that in the above-mentioned application for monitoring products in a store, the fog server 4 is set in the target store together with the camera 3. The fog server 4 is set for each store in this way, and thus the cloud server 1 does not need to directly receive the transmission data from the plurality of cameras 3, and the processing burden of the cloud server 1 is reduced.

[0388] In addition, when there are multiple target stores and all the stores belong to the same chain store, one fog server 4 may be provided for the multiple stores instead of providing a fog server 4 for each store.

[0389] In the following description, the above-mentioned various devices can be roughly divided into cloud-side information processing devices and edge-side information processing devices.

[0390] The cloud-side information processing apparatus includes the cloud server 1 and other server apparatuses, and is an apparatus group that provides services intended to be used by a plurality of users.

[0391] The edge-side information processing device includes a camera 3 , a fog server 4 , and a user terminal 2 , and can be considered as a device group provided in an environment prepared by a user who uses a cloud service.

[0392] However, both the cloud-side information processing device and the edge-side information processing device may be set in an environment prepared by the same user.

[0393] The fog server 4 may be a local server.

[0394] The fog server 4 has, for example, Figure 4 The hardware configuration shown. The communication unit 35 of the camera 3 ( Figure 3 , Fig.25 , Fig. 27 etc.) are configured to be able to communicate with Fig.34 The fog server 4 shown performs data communication.

[0395] The fog server 4 performs the processing performed by the signal processing unit 36 ​​in the first embodiment and the signal processing unit 36 ​​in the second embodiment. The fog server 4 may also implement these processing as rule-based signal processing or AI image processing.

[0396] The information processing system 100 includes the fog server 4 , and thus it is possible to adopt a product with low computing power as the camera 3 and reduce the introduction cost of the camera 3 .

[0397] <9. User Interface>

[0398] The result of the above-mentioned notification process performed by the cloud server 1 and the like can be viewed by the store manager via the application executed in the user terminal 2 .

[0399] Specifically, the cloud server 1 manages various data about products in a store, performs analysis based on the data, visualizes the results, and provides the results to an administrator. That is, the cloud server 1 has a report function and a dashboard function as a business intelligence (BI) tool.

[0400] In the user terminal 2 used by the administrator, various information including the above-mentioned notification information can be viewed by starting an application for viewing the visualized analysis results. The application started in the user terminal 2 can be a dedicated application or a web browser.

[0401] Fig.35 An example of a screen that can be viewed by an administrator in the user terminal 2 is shown in FIG. In the following description, an example of a case where the inventory status of bread is managed is given. In this example, a processing target shelf image is captured every 5 minutes.

[0402] Fig.35 The real-time stock status screen G1 is shown. The real-time stock status screen G1 is configured with an image display area 10 in which a product shelf captured by the camera 3 and shelf images of various types of bread displayed are displayed, and other areas.

[0403] Here, the image displayed in the image display area 10 is a shelf image captured by the camera 3. The camera 3 may send the shelf image to the cloud server 1 at a certain time interval, or as described above, may capture a reference shelf image before the store opens and send the reference shelf image to the cloud server 1, and then send only a partial image to the cloud server 1.

[0404] When only a partial image is sent from the camera 3 to the cloud server 1, the reference shelf image captured before the store opens is displayed in the image display area 10. When a partial image is received from the camera 3, the partial image can be superimposed on the corresponding area of ​​the reference shelf image to reproduce the real-time product shelf display situation in the store.

[0405] The real-time inventory status screen G1 is provided with a selected product information display area 11 , a zoom operation area 12 , and a time display area 13 above the image display area 10 .

[0406] The selected product information display area 11 is an area that displays the type, group information, and the like of the product (bread) selected on the image display area 10 .

[0407] Here, group information is a collection of display spaces that display the same product. For example, in Fig.35 In the state shown, the same product is displayed in three display spaces vertically adjacent to each other and linked together as a group (thick frame in the figure).

[0408] Therefore, the selected product information display area 11 displays that the selected product is a group product displayed in a plurality of display spaces, and displays the product name.

[0409] The zoom operation area 12 is an area operated when it is desired to view the display state of the product in detail, and makes it possible to enlarge or reduce the shelf image displayed in the image display area 10 .

[0410] The time display area 13 is an area for displaying the time when the shelf image displayed in the image display area 10 was captured. The manager can compare the time with the current time to estimate the status of the product shelf at the current time.

[0411] At the uppermost portion of the real-time stock status screen G1 , an image switching operator 14 for switching shelf images to be displayed in the image display area 10 , a data update operator 15 , and a display switching operator 16 are provided.

[0412] The image switching operator 14 is an operator used when there are a plurality of product shelves under management and shelf images of the product shelves are captured by a plurality of cameras 3. Fig.35 In the example shown, two product shelves are captured by two cameras 3 , and a shelf image of a second product shelf (shelf 2 ) is displayed in the image display area 10 .

[0413] The data update operator 15 is an operator for reflecting information on the real-time stock status screen G1 when an existing shelf image (or a partial image) or a notification is received after the application is started.

[0414] The display switching operator 16 is an operator for switching whether to superimpose a label indicating the stock status on each display space of the shelf image displayed in the image display area 10 . Fig.35 The state without superimposed labels is shown. Fig.36 A state in which a label is displayed superimposed on the shelf image displayed in the image display area 10 is shown.

[0415] The log information display area 17 is provided on the right side of the image display area 10 .

[0416] In the log information display area 17, the history of notifications received from the cloud server 1 is displayed. As described above, the displayed notifications include various statuses such as "out of stock status", "supply shortage status", "inappropriate posture status", "inappropriate display position status", and "inappropriate price tag status", but a filtering function of selecting and displaying only some notifications may be provided.

[0417] A timeline display area 18 is provided below the image display area 10 , and a graphic display area 19 is provided below the timeline display area 18 .

[0418] In the timeline display area 18, a time axis whose length corresponds to the opening hours of the store and a pointer 18a displayed on the time axis are displayed.

[0419] The pointer 18 a is an operator that can be slid left and right along the time axis, and is operated to determine the state of the shelf image to be displayed in the image display area 10 .

[0420] The graph display area 19 is an area for displaying the time series data of the quantity of the product selected in the image display area 10 in the form of a graph.

[0421] A memory selection area 20 and a function selection area 21 are provided on the left side of the image display area 10 .

[0422] The store selection field 20 is an operation field for selecting one from among the options so that a store such as “Shibuya store” or “Harajuku store” can be uniquely specified. The shelf image to be displayed in the image display area 10 changes according to the selection operation in the store selection area 20 .

[0423] The function selection field 21 is an operation field that can select a function as a BI tool provided by the cloud server 1. Functions that can be selected through the function selection field 21 include, for example, "inventory detection (grasping)" for detecting and visually displaying information about the inventory of products, "flow line analysis" for analyzing and displaying the movement of customers in a store, and "customer analysis" for analyzing customer demographics and genders of customers.

[0424] Two tabs are provided below the function selection field 21 , one of which is a real-time tab 22 and the other is a setting tab 23 .

[0425] The real-time tab 22 is an operator for displaying each of the above-mentioned areas and operators, and is an operator for displaying the real-time inventory status screen G1. Specifically, Fig.35 and Fig.36 The state in which the real-time tab 22 is selected is shown.

[0426] The setting tab 23 is an operator for displaying a setting screen G2 for performing various settings.

[0427] Fig.37 A state in which the setting tab 23 is operated and the setting screen G2 is displayed is shown.

[0428] The setting screen G2 is a screen for setting the product name (or product ID) of the product to be displayed on the product shelf.

[0429] A weekday setting area 24 for performing settings for product shelves on weekdays and a holiday setting area 25 for performing settings for product shelves on holidays are vertically arranged on the setting screen G2.

[0430] In the weekday setting area 24 and the holiday setting area 25, a plurality of product selection fields 26 are arranged in a matrix form.

[0431] The arrangement of the product selection fields 26 is linked to the arrangement of the display spaces on the product shelves. Fig.37 In the example shown, according to Fig.35 and Fig.36 The product selection field 26 is arranged in the arrangement of display space on the product shelf shown.

[0432] In the product selection field 26, a product name can be selected from the set product names.

[0433] The product name may be set using a product name management button 27 provided below the holiday setting area 25 .

[0434] When the product name management button 27 is pressed, the Fig.38 The product name management screen G3 is shown. The details will be described later.

[0435] The history confirmation button 28, the save button 29 and the reset button 30 are set in Fig.37 to the right of the product name management button 27 on the setting screen G2.

[0436] The setting history confirmation button 28 is an operator for restoring a previous setting state and making it possible to confirm the previous setting state.

[0437] The save button 29 is operated to save the current settings and reapply the current settings.

[0438] The reset button 30 is operated to end the setting without saving the current setting.

[0439] like Fig.38 As shown, the product name management screen G3 is provided with an area R1 for displaying the arrangement order of product names, an area R2 for inputting product names, and an area R3 for displaying the product ID of each product.

[0440] By entering the product name in area R2, you can change Fig.35 The product names that can be selected in the product selection field 26. In addition, the region R1 shows the display order of the options displayed in the product selection field 26. The display order can be changed by operating the redistribution operator Btn1 in a state where the product to be changed is selected.

[0441] The CPU 71 of the cloud server 1 functions as a display information generating unit F21 (see FIG. 21 ) for generating display information for displaying the above-mentioned various screens on the screen of the user terminal 2. Fig.39 ).

[0442] The CPU 71 of the cloud server 1 also functions as a transmission processing unit F22 that executes a transmission process for displaying the generated display information on the user terminal 2 .

[0443] The display information generated by the display information generating unit F21 is information for displaying the notification content of the notification process, the reference shelf image, and the process target shelf image on the user terminal 2 .

[0444] The display information may also be information in which information on the quantity of the displayed product is superimposed on each display space of the product in the shelf image according to the display for the display switching operator 16 (see Fig.36 ).

[0445] Furthermore, the display information includes information for displaying a state in which the same products are grouped in an identifiable manner.

[0446] The display information also includes information that makes it possible to grasp the time lapse (history) of the number of products displayed in the graph display area 19 .

[0447] <10. Registration of AI models and AI applications>

[0448] As described above, in the information processing system 100, AI image processing is performed in the camera 3 which is an edge-side information processing device, and the result information of the edge-side AI image processing (for example, the result information of image recognition processing using AI) is used in the cloud server 1 which is a cloud-side information processing device to implement advanced application functions.

[0449] Specifically, in the second embodiment, AI image processing such as obstacle detection or different area detection processing is performed by the AI ​​image processing unit 44 provided in the image sensor IS.

[0450] In addition to this, in the third embodiment, AI image processing is also performed as the recognition processing of the state of the product.

[0451] Furthermore, in the fourth embodiment, at least a part of the AI ​​image processing may be performed in the fog server 4 which is an edge-side information processing device.

[0452] Here, various methods for registering an application function in the cloud server 1 (or the fog server 4 ) as a cloud-side information processing device are conceivable.

[0453] Will see Fig.40 Describe an example.

[0454] Despite Fig.40 The fog server 4 is not shown in the figure, but the configuration may include a fog server 4. In this case, the fog server 4 may take on some edge-side functions.

[0455] A server device such as the above-described cloud server 1 is an information processing device constituting a cloud-side environment.

[0456] In addition, the camera 3 is an information processing device constituting the edge side environment. The image sensor IS provided in the camera 3 can also be regarded as an information processing device constituting the edge side environment. That is, the image sensor IS as another edge side information processing device can be regarded as installed inside the camera 3 as an edge side information processing device.

[0457] Furthermore, in the above-described example, the user terminal 2 has been described as an information processing device used by a user who uses a product management function provided by the information processing system 100. When an information processing device used by a user who uses various services provided by the information processing system 100 is regarded as the user terminal 2, there are other conceivable aspects of the user terminal 2.

[0458] For example, an information processing device used by a user who develops an application used in AI image processing is one aspect of user terminal 2 (application developer terminal 2A), an information processing device used by a user who uses an application is also one aspect of user terminal 2 (application user terminal 2B), and an information processing device used by a user who develops an AI model used in AI image processing is also one aspect of user terminal 2 (AI model developer terminal 2C). User terminal 2 used by a product manager to use an application for the above-mentioned product management may be referred to as application user terminal 2B.

[0459] The application developer terminal 2A can be used by users who develop applications that do not use AI image processing.

[0460] The cloud-side information processing device is provided with a training data set for performing AI training and an AI model used as a basis for development. The user who develops the AI ​​model uses the AI ​​model developer terminal 2C to communicate with the cloud-side information processing device and download these training data sets and AI models. At this time, the training data set can be provided for a fee. For example, an AI model developer can purchase a training data set in a state where various functions and materials registered in the market are purchased by registering personal information in a market (electronic market) prepared as a function on the cloud side.

[0461] After the AI ​​model developer develops the AI ​​model using the training data set, the AI ​​model developer registers the developed AI model in the market by using the AI ​​model developer terminal 2C. Thus, when the AI ​​model is downloaded, an incentive can be paid to the AI ​​model developer.

[0462] In addition, the user who develops the application can download the AI ​​model from the market by using the application developer terminal 2A, and develop the application using the AI ​​model. At this time, as described above, an incentive can be paid to the AI ​​model developer.

[0463] The application development user can register the developed AI application on the market by using the application developer terminal 2A. Thus, when the AI ​​application is downloaded, an incentive can be paid to the user who developed the AI ​​application.

[0464] The user who uses the AI ​​application performs the operation of deploying the AI ​​application and the AI ​​model from the market to the camera 3 as the edge-side information processing device managed by the user by using the application user terminal 2B. At this time, an incentive can be paid to the AI ​​model developer.

[0465] Thus, AI image processing can be performed on the camera 3 using the AI ​​application and the AI ​​model, and not only can images be captured but also AI image processing related to product management and customer attribute extraction as described above can be performed.

[0466] Here, the deployment of AI applications and AI models indicates the installation of AI applications and AI models in a target (device) as an execution object, so that the target as an execution object can use the AI ​​applications and AI models, in other words, so that at least a part of the program as an AI application can be executed.

[0467] Furthermore, in the camera 3 , attribute information of the customer may be extracted from the captured image captured by the camera 3 through AI image processing.

[0468] The attribute information is transmitted from the camera 3 to the cloud-side information processing device via the network 6 .

[0469] The cloud application is deployed in the cloud-side information processing device, and each user can use the cloud application via the network 6. In addition to the above-mentioned application for managing the number of products on the display, the cloud application is provided with an application for analyzing the flow line of the customer by using the customer attribute information and the captured image, etc. Such a cloud application is uploaded by the application development user, etc.

[0470] The application user can analyze the quantity of products on the display by using the application user terminal 2B to appropriately replenish the products on the display shelf.

[0471] In addition, the application user can use the cloud application for flow line analysis to perform flow line analysis on customers in his or her own store by using the application user terminal 2B, and can view the analysis results. The analysis results can be viewed by graphically presenting the flow lines of customers on a map of the store.

[0472] The results of the flow analysis can also be displayed in the form of a heat map, and the analysis results can be viewed by presenting the density of customers, etc.

[0473] This information can also be displayed in a classified manner according to the customer's attribute information.

[0474] In the cloud-side market, an AI model optimized for each user can be registered. For example, images captured by the camera 3 installed in a store managed by a specific user are appropriately uploaded and accumulated in the cloud-side information processing device.

[0475] In the cloud-side information processing device, every time a certain number of uploaded captured images are accumulated, a retraining process is performed on the AI ​​model, and a process for updating the AI ​​model and re-registering on the market is performed.

[0476] For example, a retraining process of an AI model may be selected by a user as an option in the marketplace.

[0477] For example, an AI model relearned using a dark image from camera 3 set inside a store is deployed in camera 3, and thus the recognition rate of image processing of images captured in a dark location can be improved. In addition, an AI model relearned using a bright image from camera 3 set outside a store is deployed in camera 3, and thus the recognition rate of image processing of images captured in a bright location can be improved.

[0478] That is, the application user deploys the updated AI model in the camera 3 again, and can therefore always obtain optimized processing result information.

[0479] The retraining process of the AI ​​model will be described later.

[0480] Furthermore, in the cloud-side market, it is possible to register an AI model optimized for the corresponding camera 3. For example, an AI model applied to a camera 3 that can acquire RGB images, an AI model applied to a camera 3 equipped with a distance measurement sensor that generates a distance image, and the like are envisioned.

[0481] In addition, an AI model trained using images of the vehicle and images captured in a bright environment as the AI ​​model used by the camera 3 during bright time, and an AI model trained using images captured in a dark environment as the AI ​​model used by the camera 3 during dark time can be registered in the market.

[0482] Through the retraining process, these AI models are preferably appropriately updated to AI models with improved recognition rates.

[0483] In addition, when the information uploaded from camera 3 to the cloud-side information processing device (such as a captured image) includes personal information, from the perspective of privacy protection (the above-mentioned privacy mask processing), data from which the privacy information has been deleted can be uploaded, or AI model development users and application development users can use data from which information about privacy has been deleted.

[0484] exist Fig.41 and Fig.42 The flow of the above-mentioned processing is shown in the flowchart.

[0485] The cloud-side information processing device corresponds to Figure 1 The cloud server 1 and other server devices in it.

[0486] When the AI ​​model developer views a list of data sets registered in the market and selects a desired data set by using the AI ​​model developer terminal 2C including a display unit such as an LCD or an organic EL panel, the AI ​​model developer terminal 2C sends a download request for the selected data set to the cloud-side information processing device in step S21.

[0487] In response to this, the cloud-side information processing device receives the request in step S1 and performs processing for sending the requested data set to the AI ​​model developer terminal 2C in step S2.

[0488] In step S22, the AI ​​model developer terminal 2C performs a process for receiving a data set. Therefore, the AI ​​model developer can develop an AI model using the data set.

[0489] When the AI ​​model developer completes the development of the AI ​​model and then performs an operation for registering the developed AI model in the market (for example, an operation of specifying the name of the AI ​​model, the address where the AI ​​model is located, etc.), the AI ​​model developer terminal 2C sends a request to register the AI ​​model in the market to the cloud-side information processing device in step S23.

[0490] In response to this, the cloud-side information processing device receives the registration request in step S3, and performs AI model registration processing in step S4, and thus, for example, the cloud-side information processing device can display the AI ​​model on the market. Thus, users other than the AI ​​model developer can download the AI ​​model from the market.

[0491] For example, an application developer who attempts to develop an AI application checks a list of AI models registered in the market by using the application developer terminal 2A. In response to the application developer's operation (e.g., an operation of selecting one of the AI ​​models on the market), the application developer terminal 2A sends a download request for the selected AI model to the cloud-side information processing device in step S31.

[0492] The cloud-side information processing device receives the request in step S5 and sends the AI ​​model to the application developer terminal 2A in step S6.

[0493] In step S32, the application developer terminal 2A receives the AI ​​model, thereby enabling the application developer to develop an AI application using the AI ​​model developed by another person.

[0494] When the application developer completes the development of the AI ​​application and then performs operations for registering the AI ​​application in the market (for example, operations for specifying the name of the AI ​​application, the address where the AI ​​model is located, etc.), the application developer terminal 2A sends a request to register the AI ​​application to the cloud-side information processing device in step S33.

[0495] The cloud-side information processing device receives the registration request in step S7 and registers the AI ​​application in step S8, and thus, for example, the cloud-side information processing device can display the AI ​​application on the market. Thus, users other than the application developer can select and download the AI ​​application on the market.

[0496] Fig.42 An example is shown in which a user other than the application developer selects and downloads an AI application on the market. For example, in response to an operation by a user attempting to use an AI application, the application user terminal 2B selects a destination in step S41. In this destination selection, the selected destination is sent to the cloud-side information processing device.

[0497] In response to this, the cloud-side information processing device selects an AI application according to the purpose in step S9, and selects an AI model in step S10. For example, table data in which AI applications and AI models are associated with each other according to the purpose is stored in the cloud-side information processing device so that AI applications and AI models according to the purpose can be selected.

[0498] In step S11 , the cloud-side information processing device executes a process for deploying the selected AI application and AI model. In this deployment process, the AI ​​application and AI model are sent to the camera 3 .

[0499] In response to this, in step S51, deployment processing of the AI ​​application and the AI ​​model is performed in the camera 3. Thus, AI image processing can be performed on the captured image captured in the camera 3.

[0500] In step S52, the camera 3 captures an image by performing an imaging operation. Then, in step S53, the camera 3 performs AI image processing on the captured image to obtain, for example, an image recognition result.

[0501] In step S54, the camera 3 transmits the captured image and the result information of the AI ​​image processing. In the transmission of the information in step S54, both the captured image and the result information of the AI ​​image processing may be transmitted, or only one of them may be transmitted.

[0502] The cloud-side information processing device that has received the information performs analysis processing in step S12. Through this analysis processing, for example, customer flow analysis, vehicle analysis processing for traffic monitoring, and the like are performed.

[0503] The cloud-side information processing device performs a process for presenting the analysis result in step S13. This process is implemented, for example, by a user using the above-mentioned cloud application imaging.

[0504] After the analysis result presentation process, the application user terminal 2B performs a process for displaying the analysis result on a monitor or the like in step S42.

[0505] Through the processing up to this point, the user of the AI ​​application can obtain the analysis result according to the purpose selected in step S41.

[0506] The AI ​​model may be updated to optimize images captured by the camera 3 that are managed by the application user.

[0507] For example, the camera 3 repeatedly performs the processing of steps S52 , S53 , and S54 , and thus the captured image received from the camera 3 and the result information of the AI ​​image processing are accumulated in the cloud-side information processing device.

[0508] Then, when a certain amount of information is accumulated, the cloud-side information processing device performs an update process of the AI ​​model in step S14. This process is a process of retraining the AI ​​model by providing new data to the AI ​​model.

[0509] In step S15 , the cloud-side information processing device performs processing for deploying a new updated AI model.

[0510] After this deployment process, in step S55 , a process for deploying a new AI model is performed in the camera 3 .

[0511] When the AI ​​application is also updated, the updated AI application program may be further deployed in the process of step S55 .

[0512] <11. System Functional Overview>

[0513] In the present embodiment, as a service using the information processing system 100, it is assumed that a user as a customer can select a service of a function type for AI image processing for each camera 3. The selection of the function type can be referred to as setting of the above purpose. For example, an image recognition function, an image detection function, etc. can be selected, or a more detailed type can be selected to perform an image recognition function or an image detection function for a specific object.

[0514] For example, as a business model, a service provider sells a camera 3 and a fog server 4 with an AI image recognition function to a user, and installs the camera 3 and the fog server 4 at a location to be monitored. Then, the service provider deploys a service for providing the above-mentioned analysis information to the user.

[0515] At this time, since the purpose (purpose) of systems such as product monitoring, store monitoring or traffic monitoring is different for each customer, the AI ​​image processing function of camera 3 can be selectively set so that analysis information corresponding to the purpose desired by the customer can be obtained.

[0516] In addition, it is also considered that when a disaster such as an earthquake occurs, the information expected to be acquired using the camera 3 will change. Specifically, in normal times, the AI ​​image processing function for detecting customers and specifying their attributes is activated to realize the function as a store monitoring camera, and in the event of a disaster, the AI ​​image processing function is switched to the AI ​​image processing function for grasping the remaining products on the product shelf. At the time of this switching, it is considered to change the AI ​​model so that an appropriate recognition result can be obtained.

[0517] In this example, the cloud-side information processing device has a function of selectively setting the AI ​​image processing function of the camera 3 .

[0518] The fog server 4 as an edge-side information processing device may have the function of a cloud-side information processing device.

[0519] Here, see Fig.43 The connection between the cloud server 1 as a cloud-side information processing device and the camera 3 as an edge-side information processing device is described.

[0520] The cloud-side information processing device is equipped with a retraining function, a device management function, and a market function, which are functions that can be used via a hub.

[0521] The hub performs secure and highly reliable communication with the edge information processing device, thereby providing various functions to the edge information processing device.

[0522] The retraining function is a function of performing retraining and providing a newly optimized AI model, which provides an appropriate AI model based on new training materials.

[0523] The device management function is a function of managing the camera 3 or the like as an edge-side information processing device, and can provide functions such as management and monitoring of the AI ​​model deployed in the camera 3, problem detection, and troubleshooting.

[0524] In addition, the device management function is a function of managing information about the camera 3 and the fog server 4. The information about the camera 3 and the fog server 4 includes information about a chip used as an arithmetic processing unit, memory capacity, storage capacity, information such as usage rate of a CPU and memory, and information about software such as an operating system (OS) installed in each device.

[0525] In addition, device management functions are used to protect secure access for authenticated users.

[0526] The market function provides functions for registering the above-mentioned AI models developed by AI model developers and AI applications developed by application developers, and for deploying these developments to permitted edge-side information processing devices, etc. In addition, the market function also provides functions related to payment of rewards based on deployed developments.

[0527] The camera 3 as an edge-side information processing device includes an edge runtime, an AI application, an AI model, and an image sensor IS.

[0528] The edge runtime serves as embedded software for managing applications deployed in the camera 3 and communicating with a cloud-side information processing device.

[0529] As described above, the AI ​​model is a deployed AI model registered in the market in the cloud-side information processing device, and thus the camera 3 can obtain result information of AI image processing according to the purpose by using the captured image.

[0530] Will refer to Fig.44 An overview of the functions of the cloud-side information processing device is described. The cloud-side information processing device is a general term for various server devices such as the cloud server 1 .

[0531] As shown in the figure, the cloud-side information processing device has a license authorization function F31, an account service function F32, a device monitoring function F33, a market function F34 and a camera service function F35.

[0532] The licensing function F31 is a function for executing various authentication-related processes. Specifically, in the licensing function F31, processes related to device authentication of each camera 3 and processes related to authentication of each of the AI ​​model, software, and firmware used in the camera 3 are executed.

[0533] Here, the above-mentioned software is software required to appropriately implement AI image processing in the camera 3 .

[0534] In order to properly perform AI image processing based on captured images, and to send the results of AI image processing to the fog server 4 and the cloud server 1 in an appropriate format, it is necessary to control the data input to the AI ​​model and to properly process the output data of the AI ​​model. The above-mentioned software is software that includes peripheral processing required to properly implement AI image processing. This software is software for realizing the desired function by using the AI ​​model, and corresponds to the above-mentioned AI application.

[0535] The AI ​​application is not limited to an AI application using only one AI model, but an AI application using two or more AI models can be conceived. For example, there may be an AI application having a processing flow in which image data (such as image data, hereinafter referred to as "recognition result information"), which is information about a recognition result obtained by an AI model that performs AI image processing using a captured image as an input tensor, is input as an input tensor to another AI model to perform a second AI image processing.

[0536] Alternatively, the AI ​​application may be an AI application that performs a predetermined image processing as a second AI image processing on an input tensor for the first AI image processing by using the coordinate information as the recognition result information of the first AI image processing. The input tensor for each AI image processing may be an original image, an RGB image obtained by performing synchronization processing on the original image, or the like. The same applies to the following description.

[0537] In the license granting function F31 , when the camera 3 is connected to the network 6 , a process for issuing a device ID of each camera 3 is performed to authenticate the camera 3 .

[0538] In addition, when the AI ​​model and software are authenticated, for each AI model and AI application registered from the AI ​​model developer terminal 2C and the software developer terminal 7, a process of issuing a unique ID (AI model ID, software ID) for each AI model and AI application is performed.

[0539] In addition, in the license granting function F31, processing is performed for issuing various keys and certificates for performing secure communication between the camera 3, the AI ​​model developer terminal 2C, the software developer terminal 7 and the cloud server 1 to the manufacturer of the camera 3 (specifically, the manufacturer of the image sensor IS described below), the AI ​​model developer and the software developer, and processing is also performed for updating or stopping the validity of the certificate.

[0540] Furthermore, in the licensing function F31, when user registration (registration of account information accompanied by issuance of a user ID) is performed by the account service function F32 described below, processing for associating the camera 3 (the above-mentioned device ID) purchased by the user with the user ID is also performed.

[0541] The account service function F32 is a function for generating and managing the user's account information. In the account service function F32, input of user information is received, and account information is generated based on the input user information (account information including at least user ID and password information is generated).

[0542] In addition, in the account service function F32, registration processing (registration of account information) of AI model developers and AI application developers (hereinafter may be simply referred to as "software developers") is also performed.

[0543] The device monitoring function F33 is a function for executing a process for monitoring the use status of the camera 3. For example, as various factors related to the use status of the camera 3 (such as the position where the camera 3 is used), the output frequency of output data of the AI ​​image processing, and the free capacity of the CPU and memory used for the AI ​​image processing, information such as the use rate of the above-mentioned CPU and memory is monitored.

[0544] The market function F34 is a function for selling AI models and AI applications. For example, users can purchase AI applications and AI models used by AI applications via a sales website (sales site) provided by the market function F34. Software developers can also purchase AI models used to create AI applications via the sales site.

[0545] The camera service function F35 is a function for providing the user with services related to the use of the camera 3 .

[0546] An example of the camera service function F35 is the above-mentioned report function or dashboard function. That is, this function is a function for generating analysis information about an object based on the processing result information of the image processing in the camera 3 and allowing the user to view the analysis information via the user terminal 2.

[0547] In addition, the camera service function F35 includes an imaging setting search function. Specifically, the imaging setting search function is a function of acquiring recognition result information of AI image processing from the camera 3, and based on the acquired recognition result information, searching the imaging setting information of the camera 3 using AI. Here, the imaging setting information broadly refers to setting information related to the imaging operation for obtaining an image. Specifically, it widely includes optical settings such as focus and aperture, settings related to the readout operation of the captured image signal (such as frame rate, exposure time and gain), and further settings related to the image signal processing of the readout captured image signal (such as gamma correction processing, noise reduction processing and super-resolution processing).

[0548] When the imaging setting search function is appropriately performed, the imaging setting of the camera 3 is optimized according to the purpose set by the user, and a satisfactory inference result can be obtained.

[0549] The camera service function F35 also includes an AI model search function. The AI ​​model search function obtains the recognition result information of the AI ​​image processing from the camera 3, and based on the obtained recognition result information, uses AI to search for the best AI model to be used for the AI ​​image processing in the camera 3. The AI ​​model search here refers to the process of optimizing various processing parameters related to the neural network structure, such as weight coefficients and setting information (for example, including kernel size information), when AI image processing is implemented by a convolutional neural network (CNN) including a convolution operation.

[0550] The camera service function F35 may include a function of determining a processing burden. In the processing burden determination function, when an AI application is deployed in an edge-side information processing device, the process of the deployment destination device is determined in units of SW components. Some SW components may be determined to be executed in a cloud-side device, and in this case, the deployment process may not be performed because the application has already been deployed in the cloud-side device.

[0551] By having the imaging setting search function and the AI ​​model search function as described above, imaging settings that produce satisfactory AI image processing results can be performed, and AI image processing can be performed by using an appropriate AI model according to the actual use environment.

[0552] In addition, by having a processing load determination function, AI image processing and its analysis processing can be executed in an appropriate device.

[0553] The camera service function F35 has an application setting function before deploying each SW component. The application setting function is a function to set an appropriate AI application according to the user's purpose.

[0554] For example, an appropriate AI application is selected according to the purpose selected by the user. Thus, the SW components constituting the AI ​​application are automatically determined. There may be a combination of multiple types of SW components for achieving the user's purpose by using the AI ​​application, and in this case, one combination is selected in response to the information about the edge-side information processing device and the user's request.

[0555] For example, when a user aims to monitor a store, the combination of SW components may differ depending on whether the user's request places importance on privacy or speed.

[0556] In the application setting function, the user terminal 2 (corresponding to Fig.40 The processing for receiving the user's selection of the purpose (application) on the application user terminal 2B), the processing for selecting an appropriate AI application based on the selected application, etc.

[0557] Here, the above example illustrates a configuration in which the license authorization function F31, the account service function F32, the device monitoring function F33, the market function F34, and the camera service function F35 are implemented by a single cloud server 1, but a configuration in which these functions are divided and implemented by multiple information processing devices may also be adopted. For example, it is conceivable that each of the above functions is performed by one information processing device. Alternatively, a single function of the above functions may also be performed by multiple information processing devices (e.g., cloud server 1).

[0558] Various configurations of the image sensor IS are conceivable. Here, an example is described in which the image sensor IS has a two-layer structure.

[0559] like Fig.45 As shown, the image sensor IS is configured as a single-chip semiconductor device in which two dies are laminated.

[0560] The image sensor IS is configured such that a die D1 and a die D2 are laminated to each other, and the die D1 has Figure 3 , Fig.25 and Fig. 27 The functions of the imaging unit 41 shown are shown, and the tube die D2 includes an image signal processing unit 42, a sensor internal control unit 43, an AI image processing unit 44, a memory unit 45 and a communication I / F 46.

[0561] The die D1 and the die D2 are electrically connected, for example, by Cu-Cu bonding.

[0562] <12. Deployment of AI models and AI applications>

[0563] Various methods of deploying AI models and AI applications in the camera 3 are envisioned. As an example, an example using container technology will be described.

[0564] In the camera 3, the operating system 51 is installed in the Figure 3 The various hardware 50 of the control unit 33 shown in FIG. 3 such as a CPU, a graphics processing unit (GPU), a ROM, and a RAM are provided (see FIG. Fig.46 ).

[0565] The operating system 51 is basic software that performs overall control of the camera 3 to implement various functions in the camera 3 .

[0566] The general middleware 52 is installed on the operating system 51 .

[0567] The general-purpose middleware 52 is software for realizing basic operations such as a communication function using the communication unit 35 as the hardware 50 and a display function using a display unit (monitor, etc.) as the hardware 50 .

[0568] In addition to the general middleware 52 , an orchestration tool 53 and a container engine 54 are also installed on the operating system 51 .

[0569] The orchestration tool 53 and the container engine 54 deploy and execute the container 55 by building a cluster 56 as an operating environment for the container 55 .

[0570] Fig.43 The edge runtimes shown correspond to Fig.46 An orchestration tool 53 and a container engine 54 are shown.

[0571] The orchestration tool 53 has a function of appropriately allocating resources of the above-mentioned hardware 50 and operating system 51 to the container engine 54. The orchestration tool 53 groups containers 55 into predetermined units (pods described later), and the pods are deployed in worker nodes (described later) which are logically different areas.

[0572] The container engine 54 is one of the middleware installed in the operating system 51, and is an engine that operates the container 55. Specifically, the container engine 54 has a function of allocating resources (memory, computing power, etc.) of the hardware 50 and the operating system 51 to the container 55 based on the setting file of the middleware in the container 55, etc.

[0573] Furthermore, in the present embodiment, the resources to be allocated include not only the resources of the control unit 33 of the camera 3 but also the resources of the sensor internal control unit 43, the memory unit 45, and the communication I / F 46 of the image sensor IS.

[0574] The container 55 is configured to include middleware such as an application program and a library for realizing a predetermined function.

[0575] The container 55 operates to implement a predetermined function using resources of the hardware 50 and the operating system 51 allocated by the container engine 54 .

[0576] In this embodiment, Fig.43 The AI ​​application and the AI ​​model shown correspond to one of the containers 55. That is, one of the various containers 55 deployed in the camera 3 implements a predetermined AI image processing function using the AI ​​application and the AI ​​model.

[0577] Will refer to Fig.47 A specific example of the configuration of the cluster 56 constructed by the container engine 54 and the orchestration tool 53 is described. The cluster 56 can be constructed across multiple devices so that functions are implemented using not only the hardware 50 of one camera 3 but also resources of other hardware of other devices.

[0578] The orchestration tool 53 manages the execution environment of the container 55 in units of the worker node 57. The orchestration tool 53 also constructs a master node 58 that manages the entire worker node 57.

[0579] A plurality of pods 59 are deployed in the worker node 57. The pod 59 is configured to include one or more containers 55 and implement a predetermined function. The pod 59 is regarded as a management unit for managing the container 55 by the orchestration tool 53.

[0580] The operation of the pod 59 in the working node 57 is controlled by the pod management library 60.

[0581] The pod management library 60 is configured to include a container runtime for allowing the pods 59 to use the resources of the logically allocated hardware 50, an agent controlled by the master node 58, a network agent that communicates with the pods 59 and with the master node 58, etc.

[0582] That is, the pod 59 can use each resource through the pod management library 60 to implement a predetermined function.

[0583] The master node 58 is configured to include an application server 61 that deploys the container 59, an administrator 62 that manages the deployment status of the application server 61 on the container 59, a scheduler 63 that determines the worker node 57 on which the container 55 is deployed, and a data sharing unit 64 that shares data.

[0584] By using Fig.46 and Fig.47 The configuration shown can deploy the above-mentioned AI applications and AI models in the image sensor IS of camera 3 using container technology.

[0585] As mentioned above, AI models can be Figure 3The communication I / F 46 and the like in the image sensor IS are stored in the memory unit 45 in the image sensor IS, and the AI ​​image processing can be performed in the image sensor IS, or the memory unit 45 in the image sensor IS and the sensor control unit 43 can be deployed. Fig.46 and Fig.47 The configuration shown, and the above-mentioned AI applications and AI models can be executed in the image sensor IS using container technology.

[0586] In addition, as described below, container technology can also be used when deploying AI applications and / or AI models in the fog server 4 or the cloud-side information processing device.

[0587] In this case, information about the AI ​​application or AI model is deployed in a container such as Figure 4 The nonvolatile storage unit 74, the storage unit 79 or the memory of the RAM 73 is stored in the nonvolatile storage unit 74, the storage unit 79 or the memory of the RAM 73 and is then executed.

[0588] <13. Others>

[0589] As mentioned above, reference Fig.48 A processing flow is described in which, after the SW components of the AI ​​application and the AI ​​model are deployed, the operation of the service provider or the user is used as a trigger to perform retraining of the AI ​​model and update of the AI ​​model deployed in the camera 3, etc. (hereinafter referred to as the "edge-side AI model") and the AI ​​application. Fig.48 It is shown by focusing on one camera 3 among the multiple cameras 3. In addition, in the following description, the edge-side AI model to be updated is deployed in the image sensor IS of the camera 3 as an example, but the edge-side AI model can be naturally deployed outside the image sensor IS in the camera 3.

[0590] First, in processing step PS1, the service provider or user issues an instruction to relearn the AI ​​model. The instruction is issued using an API function of an application programming interface (API) module included in the cloud-side information processing device. The instruction also specifies the amount of images used for training (e.g., the number of images). Hereinafter, the amount of images used for training will also be referred to as the "predetermined number of images".

[0591] The API module receives the indication and sends a signal to the hub (with Fig.43 Similar to what is shown) sends a request for retraining and information about the amount of images.

[0592] In processing step PS3, the hub transmits an update notification and information on the amount of images to the camera 3 which is an edge-side information processing device.

[0593] In process step PS4, the camera 3 transmits the captured image data obtained by performing imaging to the image database (DB) of the storage group. This imaging process and transmission process are performed until a predetermined number of images required for retraining is reached.

[0594] When the camera 3 obtains an inference result by performing inference processing on the captured image data, the inference result may be stored in the image DB as metadata of the captured image data in processing step PS4.

[0595] The inference results in the camera 3 are stored as metadata in the image DB, and thus the data required for retraining the AI ​​model executed on the cloud side can be carefully selected. Specifically, retraining can be performed using only image data whose inference results in the camera 3 are different from the inference results executed in the cloud side information processing device using abundant computer resources. Therefore, the time required for retraining can be shortened.

[0596] After capturing and sending a predetermined number of images, in process step PS5, the camera 3 notifies the hub that the sending of the predetermined number of captured image data has been completed.

[0597] Upon receiving the notification, the hub notifies the orchestration tool 53 that the preparation of data for retraining has been completed in process step PS6.

[0598] In process step PS7, the orchestration tool 53 sends an instruction to the marking module to perform the marking process.

[0599] The labeling module acquires the image data to be subjected to labeling processing from the image DB (processing step PS8), and executes the labeling processing.

[0600] The labeling processing mentioned here may be processing for performing the above-mentioned category recognition, may be processing for estimating the gender and age of the object of the image and assigning a label, may be processing for estimating the posture of the object and assigning a label, or may be processing for estimating the action of the object and assigning a label.

[0601] The labeling process may be performed manually or automatically. The labeling process may be completed by an information processing device on the cloud side, or may be realized by using a service provided by another server device.

[0602] After completing the labeling process, the labeling module stores the labeling result information in the dataset DB in processing step PS9. Here, the information stored in the dataset DB may be a combination of label information and image data, or may be image ID information for specifying image data rather than the image data itself.

[0603] In processing step PS10, the storage management unit that detects that the marking result information has been stored notifies the layout tool 53.

[0604] The layout tool 53 that has received the notification confirms that the labeling process for the predetermined number of image data has been completed, and sends a retraining instruction to the retraining module in the processing step PS11.

[0605] The retraining module that receives the retraining instruction acquires the data set to be used for training from the data set DB in processing step PS12 , and acquires the AI ​​model to be updated from the trained AI model DB in processing step PS13 .

[0606] The retraining module uses the acquired data set and AI model to relearn the AI ​​model. In processing step PS14, the updated AI model obtained in this way is stored in the relearned AI model DB.

[0607] In processing step PS15 , when detecting that an updated AI model has been stored, the storage management unit notifies the orchestration tool 53 .

[0608] In process step PS16, orchestration tool 53 receives the notification and sends an instruction to convert the AI ​​model to the conversion module.

[0609] In processing step PS17 , the conversion module that has received the conversion instruction acquires the updated AI model from the trained AI model DB and performs conversion processing on the AI ​​model.

[0610] In the conversion process, the conversion process is performed according to the specification information of the camera 3 as the device at the deployment destination. In this process, miniaturization is performed to minimize the loss of performance of the AI ​​model, and file format conversion is performed so that the AI ​​model can operate on the camera 3.

[0611] The AI ​​model converted by the conversion module is the edge-side AI model described above. In processing step PS18, the converted AI model is stored in the converted AI model DB.

[0612] In processing step PS19 , when a storage management unit that has stored the converted AI model is detected, the storage management unit notifies the orchestration tool 53 .

[0613] In process step PS20, orchestration tool 53 having received the notification sends a notification for updating the AI ​​model to the hub. The notification includes information for specifying a location for storing the AI ​​model for updating.

[0614] The hub that receives the notification sends an instruction to update the AI ​​model to the camera 3. The update instruction also includes information for specifying a location where the AI ​​model is stored.

[0615] In the processing step PS22 , the camera 3 performs a process for acquiring a target converted AI model from the converted AI model DB and deploying it, thereby updating the AI ​​model used in the image sensor IS of the camera 3 .

[0616] In processing step PS23 , the camera 3 that has completed updating of the AI ​​model by deploying the expanded AI model transmits an update completion notification to the hub.

[0617] In processing step PS24, the hub that has received the notification notifies the orchestration tool 53 that the AI ​​model update process for the camera 3 has been completed.

[0618] Here, the image sensor IS (eg, Figure 3 45) shown in the figure, but even if the storage unit (e.g. Figure 3 When the AI ​​model is deployed and used in a storage unit within the fog server 4 or a storage unit within the fog server 4, the AI ​​model can also be updated in the same manner.

[0619] In this case, when the AI ​​model is deployed, the device (location) where the AI ​​model is deployed is stored in a storage management unit on the cloud side, etc., and the hub reads out the device (location) where the AI ​​model is deployed from the storage management unit and sends an update indication for the AI ​​model to the device where the AI ​​model is deployed.

[0620] In processing step PS22, the device that receives the update instruction obtains the target converted AI model from the converted AI model DB and deploys it, thereby updating the AI ​​model of the device that receives the update instruction.

[0621] When only the AI ​​model is updated, the process is now complete.

[0622] When an AI application using an AI model is updated in addition to the AI ​​model, the processing to be described below is performed.

[0623] Specifically, in processing step PS25, orchestration tool 53 sends a download instruction for an AI application (such as updated firmware) to the deployment control module.

[0624] In process step PS26, the deployment control module sends an instruction to deploy the AI ​​application to the hub. The instruction includes information for specifying a location to store the updated AI application.

[0625] In process step PS27 , the hub sends a deployment instruction to camera 3 .

[0626] In process step PS28 , camera 3 downloads the updated AI application from the container DB of the deployment control module and deploys it.

[0627] In the above description, an example of sequentially performing the update of the AI ​​model operating on the image sensor IS of the camera 3 and the update of the AI ​​application operating outside the image sensor IS of the camera 3 has been described.

[0628] Here, in order to simplify the description, the AI ​​application has been described, but as described above, the AI ​​application is defined by multiple SW components such as SW components B1, B2, B3, ..., and Bn. When the AI ​​application is deployed, the cloud-side storage management unit stores where each SW component is deployed. When the hub performs processing step PS27, it reads out the device (location) where each SW component is deployed from the storage management unit and sends a deployment instruction to the device that deploys the SW component. In processing step PS28, the device that has received the deployment instruction downloads the updated SW component from the container DB of the deployment control module and deploys it.

[0629] The AI ​​applications mentioned here are SW components that are different from AI models.

[0630] In addition, when it is assumed that both the AI ​​model and the AI ​​application are operated on one device, both the AI ​​model and the AI ​​application can be updated together as a container. In this case, the update of the AI ​​model and the update of the AI ​​application can be performed simultaneously instead of sequentially. This can be achieved by executing processing steps PS25, PS26, PS27 and PS28.

[0631] For example, when containers for both the AI ​​model and the AI ​​application can be deployed in the image sensor IS of the camera 3, the AI ​​model and the AI ​​application can be updated by performing the processing steps PS25, PS26, PS27 and PS28 as described above.

[0632] By performing the above-mentioned processing, the AI ​​model is relearned using the captured image data captured in the user's usage environment. Therefore, an edge-side AI model that can output highly accurate recognition results in the user's usage environment can be generated.

[0633] In addition, even when the imaging environment of camera 3 changes, such as when a vehicle equipped with vehicle-mounted camera 3 is traveling in an area different from before, or when the amount of light incident on the imaging device changes due to changes in weather or time, the AI ​​model can be appropriately relearned each time so that the recognition accuracy of the AI ​​model can be maintained without reducing the recognition accuracy.

[0634] The above processing can be performed not only when retraining the AI ​​model, but also when operating the system for the first time in the user's usage environment.

[0635] <14. Example of market screen>

[0636] An example of a screen of a market presented to a user will be described with reference to the drawings.

[0637] Fig.49 An example of the login screen G11 is shown.

[0638] The login screen G11 includes an ID input area 91 for inputting a user ID and a password input area 92 for inputting a password.

[0639] A login button 93 for executing login and a cancel button 94 for canceling login are provided below the password input area 92 .

[0640] Furthermore, below the button, an operator for switching to a page for users who have forgotten their passwords, an operator for switching to a page for new user registration, and the like are appropriately provided.

[0641] When the login button 93 is pressed after inputting an appropriate user ID and password, a process for transitioning to a user-specific page is performed on each of the cloud server 1 and the user terminal 2 .

[0642] Fig.50 An example of a screen presented to, for example, an AI application developer using the application developer terminal 2A or an AI model developer using the AI ​​model developer terminal 2C is shown.

[0643] Developers can purchase training data sets, AI models, and AI applications for development through the market. In addition, developers can register their own AI applications and AI models in the market.

[0644] exist Fig.50 In the developer screen G12 shown, purchasable training data sets, AI models, AI applications, etc. (hereinafter collectively referred to as "data") are displayed on the left.

[0645] Although not shown in the figure, when a training data set is purchased, training can be prepared by displaying an image of the training data set on a display, surrounding only a desired portion of the image with a frame using an input device such as a mouse, and inputting only a name.

[0646] For example, when it is desired to perform AI training using an image of a cat, the image annotated with a cat can be prepared for AI training by surrounding only a portion of the cat on the image using a frame and inputting "cat" as a text input.

[0647] In addition, in order to find desired data more easily, a purpose such as "traffic monitoring", "flow line analysis" or "customer counting" can be selected. In other words, display processing is performed in each of the cloud server 1 and the user terminal 2 in which only data matching the selected purpose is displayed.

[0648] In the developer screen G12, the purchase price of each data may be displayed.

[0649] In addition, on the right side of the developer screen G12, an input field 95 for registering a training data set collected or created by the developer and an AI model and an AI application developed by the developer is provided.

[0650] For each data, an input field 95 for inputting the name and storage location of the data is provided. In addition, for the AI ​​model, a check box 96 for setting whether retraining is required is provided.

[0651] A price setting field (shown as an input field 95 in the drawing) for setting a price required to purchase data to be registered, etc. may be provided.

[0652] In addition, the user name, the last login date, etc. can be displayed as part of the user information in the upper part of the developer screen G12. In addition, the amount of currency and points that the user can use when purchasing data can also be displayed.

[0653] Fig.51 An example of a user screen G13 presented to a user (the above-mentioned application user) who performs various analyses by deploying an AI application or an AI model in the camera 3 as an edge-side information processing device managed by a user is shown.

[0654] The user can purchase the camera 3 to be deployed in the space to be monitored via the market. Therefore, radio buttons 97 allowing the user to select the type and performance of the image sensor IS installed in the camera 3 and the performance of the camera 3 are provided on the left side of the user screen G13.

[0655] The user can also purchase an information processing device as the fog server 4 via the market. Thus, radio buttons 97 that allow the user to select each performance of the fog server 4 are provided on the left side of the user screen G13.

[0656] In addition, a user who already has a fog server 4 can register the performance of the fog server 4 by inputting the performance information of the fog server 4 here.

[0657] The user can realize the desired function by installing the purchased camera 3 (or the camera 3 purchased without going through the market) in any place (such as a store) managed by him or her. In the market, the user can register the information about the location where the camera 3 is installed so as to maximize the function of each camera 3.

[0658] Radio buttons 98 are provided on the right side of the user screen G13 for allowing the user to select environmental information about the environment in which the camera 3 is installed. The user can set the above-described optimal imaging setting of the target camera 3 by appropriately selecting environmental information about the environment in which the camera 3 is installed.

[0659] When purchasing a camera 3 and the location of the camera 3 to be purchased has been determined, by selecting each item on the left side and each item on the right side of the user screen G13, it is possible to purchase a camera 3 with optimal imaging settings already set according to the planned installation location.

[0660] In the user screen G13, an execution button 99 is set. By pressing the execution button 99, the screen switches to a confirmation screen for confirming the purchase and a confirmation screen for confirming the setting of the environmental information. Thus, the user can purchase the desired camera 3 and fog server 4 and set the environmental information of the camera 3.

[0661] In the market, when the installation location of the camera 3 is changed, the environmental information of each camera 3 can be changed. By inputting the environmental information of the installation location of the camera 3 again in the change screen (not shown), the optimal imaging setting of the camera 3 can be set again.

[0662] <15. Conclusion>

[0663] As described above in each example, the sending device (camera 3 or fog server 4) includes: an image acquisition unit F1, which acquires a first shelf image (reference shelf image) that images a product shelf on which products are displayed at a first time (for example, a predetermined time before the store opens), and a second shelf image (processing target shelf image) that images the product shelf at a second time after the first time (for example, a predetermined time after the store opens); an area designation unit F3, which designates a difference area between the first shelf image and the second shelf image; a cutout processing unit F4, which cuts out a partial image from the second shelf image so as to include the difference area; and a sending processing unit F5, which sends the first shelf image and the partial image to the receiving device (cloud server 1).

[0664] In order to issue various alarms according to the status of the products displayed on the product shelf, the transmitting device (camera 3, etc.) transmits the image of the product shelf to the receiving device (cloud server 1). The receiving device then performs image recognition processing using the received image to identify the out-of-stock status of the product, the inappropriate display position status, etc. At this time, the transmission data is reduced by transmitting only the difference area between the first shelf image and the second shelf image.

[0665] Therefore, the communication volume and power consumption related to the communication can be reduced.

[0666] As mentioned above Figure 5 As described in , the area specifying unit F3 of the camera 3 as the transmitting device can specify the position information of the different area in the second shelf image (processing target shelf image), and the transmission processing unit F5 can transmit the position information.

[0667] Thereby, the area to be compared with the partial image can be specified in the first shelf image (reference shelf image). Therefore, the state of the product can be appropriately grasped.

[0668] As mentioned above Figure 6 As described in, for the camera 3 as a transmitting device, the above-mentioned position information may be coordinate information in the second shelf image (processing target shelf image).

[0669] The coordinate information may be, for example, the X and Y coordinates of the image. Thus, the area to be compared with the partial image may be specified in the first shelf image (reference shelf image).

[0670] As mentioned above Figure 7 As described in , etc., in the camera 3 as the transmitting device, the above-mentioned position information may be information (display space ID) for specifying a position for displaying a product included in the difference area.

[0671] For example, a unique shelf ID (such as shelf T1, T2, ...) is assigned to each level of the shelf from the previous level. In addition, when multiple types of products are displayed on one level, a unique display space ID (such as display space Sp1, display space Sp2, ...) is assigned to each display space.

[0672] Then, the area specifying unit F3 specifies the shelf ID and display space ID in which the product included in the difference area is displayed. The sending processing unit F5 sends the shelf ID and display space ID together with the partial image to the receiving device (cloud server 1). Thus, in the receiving device, the image area to be compared with the partial image in the first shelf image (reference shelf image) can be specified.

[0673] As mentioned above Figure 5As described in , the transmission processing unit F5 of the camera 3 as the transmitting device can compress the first shelf image (reference shelf image) and the partial image and transmit them to the receiving device (cloud server 1).

[0674] This can further reduce the amount of transmitted data, and further reduce the amount of communication and power consumption.

[0675] As mentioned above Figure 5 As described in , when it is determined that occlusion of a product displayed on a product shelf has occurred in the first shelf image (reference shelf image), the image acquisition unit F1 of the camera 3 as a transmitting device may acquire the first shelf image again.

[0676] Thereby, the partial image can be appropriately compared with the specific area of ​​the first shelf image, and the status of the product can be reliably grasped.

[0677] As mentioned above Figure 3 As described in , the camera 3 as a sending device may include an imaging unit 41 that captures a first shelf image (reference shelf image) and a second shelf image (processing target shelf image), and the imaging unit 41 may capture the second shelf image at predetermined time intervals.

[0678] The transmitting device also serves as an imaging device. Then, the transmitting device captures a second shelf image at a predetermined time interval (such as every two hours). Thus, the number of products on display during the business hours of the store can be grasped, and the loss of sales opportunities due to insufficient inventory of products on the product shelf can be suppressed.

[0679] As mentioned above Figure 3 and Figure 5 As described in , etc., the camera 3 as the transmitting device includes a battery (power supply unit 38) that supplies a power supply voltage, and the imaging unit 41 can change the predetermined time period according to the remaining battery power of the battery.

[0680] In the transmitting device as a battery-powered imaging device (camera 3), it is desirable to optimize the imaging interval according to the remaining battery power of the battery. Thus, the imaging cycle can be extended and the situation where the second shelf image (processing target shelf image) cannot be captured due to battery exhaustion can be avoided.

[0681] As described above in each example, the cloud server 1 as a receiving device includes: a receiving processing unit F11, which receives a partial image of a second shelf image, which is cut out to include a difference area between a first shelf image (reference shelf image) captured of a product shelf on which products are displayed at a first time (for example, a predetermined time before the store opens) and a second shelf image (processing target shelf image) captured at a second time after the first time (for example, a predetermined time after the store opens); an image recognition processing unit F13, which performs image recognition processing on the partial image to identify the status of the product; and a notification processing unit F14, which performs notification processing according to the status of the product.

[0682] The receiving device receives a partial image of the second shelf image, and thus the received data can be reduced compared to receiving the entire second shelf image. By appropriately specifying the changed portion in the first shelf image, and by recognizing the state of the product using the partial image of the second shelf image, appropriate notification processing such as an alarm can be performed. Therefore, the display state of the product can be optimized and sales opportunities can be increased to obtain profits.

[0683] As mentioned above Fig.11 , Fig.12 and Fig.13 As described in, for example, the status of a product recognized by the cloud server 1 as a receiving device may be a status related to the quantity of the product.

[0684] The status related to the quantity of the product is, for example, an "out of stock status" where there is no product on the display, a "short supply status" where the quantity of the product is less than a predetermined quantity, etc. By properly identifying these product statuses, the products on the product shelves can be quickly replenished and the loss of sales opportunities can be avoided.

[0685] As mentioned above Fig.11 and Fig.14 As described in, for example, the state of the product recognized by the cloud server 1 as a receiving device may be a state regarding the posture of the displayed product.

[0686] The state of the posture of the product may be, for example, a state in which the product is down or a state in which the back of the product faces forward. When such a state is recognized, a notification is issued, and thus employees or the like can arrange the posture of the product on the product shelf, thereby maintaining a display state with a good appearance and making it possible to stimulate the desire of customers to buy and increase sales opportunities.

[0687] As mentioned above Fig.11 and Fig.15 As described, the status of the product recognized by the cloud server 1 as the receiving device may be a status regarding a display location.

[0688] The status regarding the display position of a product is, for example, a status in which a product is displayed in a different display space. For example, such a status occurs when a customer returns a product he or she picked up to a display space different from the one in which it was originally displayed when it was returned to the product shelf. A notification regarding such a product status is issued, and thus the product can be moved to an appropriate display space and the possibility that the product status is misunderstood as an out-of-stock status is eliminated. Therefore, the loss of product sales opportunities can be prevented.

[0689] As mentioned above Fig.11 and Fig.16 As described in the examples above, the status of a product recognized by the cloud server 1 as a receiving device may be the status of a price tag of the product.

[0690] For example, in a display space displaying a product, a sign (price tag) having a product name and a product price may be installed. When the sign is installed by mistake or when a different product is returned to the display space where the product is out of stock, the combination of the product and the price displayed on the sign may become inappropriate.

[0691] In this case, a process for issuing a notification that the price tag of the product is inappropriate is performed, and thus work for returning the product to an appropriate location or a brand correction work is performed, so that the loss of product sales opportunities can be prevented. Specifically, when a brand with a price higher than the original price is installed, the product price is perceived as being quite high, reducing the willingness to buy. This situation can be avoided by performing the above work.

[0692] As mentioned above Fig. 9 and Fig.18 As described in the above, the second shelf image (processing target shelf image) is captured at a predetermined time interval, and for the cloud server 1 as a receiving device, the notification processing unit F14 performs notification processing on each product and each type of notification content, and a counting processing unit F15 and an instruction processing unit F16 can be set. The counting processing unit counts the number of times the notification processing is performed for each product and each type of notification content, and the instruction processing unit issues an instruction to change the second shelf image for a predetermined time period according to the counted number of times.

[0693] For example, when the second shelf image is captured and the associated image recognition processing is performed four times during business hours and notification processing about the out-of-stock status is performed all four times, in order to appropriately replenish products, it is appropriate to capture images and perform image recognition more frequently.

[0694] With this configuration, the imaging times and notification processing times can be acquired, and the imaging times and image recognition times can be changed accordingly. Therefore, this leads to appropriate grasp of the product status and can avoid the loss of sales opportunities.

[0695] As mentioned above Fig.35 , Fig.36 , Fig.37 and Fig.38 As described in the examples above, the cloud server 1 as a receiving device may include: a display information generating unit F21 for generating display information of the notification content and the second shelf image (processing target shelf image) of the display notification processing; and a sending processing unit F22 for sending the display information.

[0696] For example, the manager of a store in which a product shelf is installed can receive web page information as display information showing the second shelf image and notification content on the management terminal. Then, by confirming the web page information on the management terminal (user terminal 2, application user terminal 2B), it is possible to grasp the products that need to be replenished or the products whose display status needs to be arranged. Therefore, it is possible to standardize the product status and increase product sales opportunities. In addition, based on the notification content, it is possible to grasp not only what products may sell well, but also what products may be returned to the product shelf after being picked up, etc. By utilizing such information in the sales strategy of the product, it can be used for preferred product placement, preferred display status, etc.

[0697] As mentioned above Fig.36 As described in the examples above, the display information sent by the cloud server 1 as a receiving device may be information in which information about the quantity of displayed products is superimposed on each display area (each display space) of each product in the second shelf image (processing target shelf image).

[0698] This makes it possible to separately identify which product is in what state and take quick action.

[0699] As above reference Fig.35 and Fig.36 As described, when the same product is displayed in different display parts (display spaces) on a product shelf, the display information transmitted by the cloud server 1 as a receiving device may include information for identifying group information of the display parts that group and display the same product.

[0700] Thus, the number of products displayed for each product can be correctly grasped. Therefore, when there is an empty space on a product shelf in a stock-out state, when the same product exists in another display space, it is possible to prevent the product from being randomly replenished and suppress unnecessary work.

[0701] As above reference Fig.35 and 36 As described, the display information transmitted by the cloud server 1 as a receiving device may include historical information on the quantity of displayed products.

[0702] Thus, product sales can be grasped in time series, which can be used for hourly sales strategies.

[0703] The information processing method in the present technology causes a computer device as a camera 3 to perform the following processing: processing for acquiring a first shelf image captured at a first time for a product shelf on which products are displayed and a second shelf image captured at a second time for the product shelf after the first time, processing for specifying a difference area between the first shelf image and the second shelf image, processing for cutting out a partial image from the second shelf image so as to include the difference area, and processing for sending the first shelf image and the partial image to a receiving device.

[0704] In addition, the information processing method in the present technology enables a computer device serving as a cloud server 1 to perform the following processing: processing for receiving a partial image of a second shelf image, the second shelf image being cut out to include a difference area between a first shelf image captured at a first time for a product shelf on which a product is displayed and a second shelf image captured at a second time after the first time, processing for performing image recognition processing on the partial image to identify the status of the product, and notification processing according to the status of the product.

[0705] The program executed by the above-mentioned information processing device (camera 3, cloud server 1) can be pre-recorded on a hard disk drive (HDD) as a recording medium built into a device such as a computer device or a ROM in a microcomputer including a CPU. Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a floppy disk, a compact disk read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a floppy disk (Blu-ray Disc (registered trademark)), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium can be provided as so-called packaged software.

[0706] In addition, such a program can be installed in a personal computer or the like from a removable recording medium, or can also be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0707] The effects described in this specification are merely examples and are not restrictive, and other effects may also be obtained.

[0708] Furthermore, the above-described examples can be combined in any manner, and even when various combinations are used, the above-described various operational effects can be obtained.

[0709] <16. This technology>

[0710] The present technology can adopt the following configurations. (1)

[0712] A sending device, comprising:

[0713] An image acquisition unit, acquiring a first shelf image captured at a first time for a product shelf on which products are displayed and a second shelf image captured at a second time for the product shelf after the first time;

[0714] an area specifying unit that specifies a difference area between the first shelf image and the second shelf image;

[0715] a cut-out processing unit that cuts out a partial image from the second shelf image so as to include a different area; and

[0716] The sending processing unit sends the first shelf image and the partial image to the receiving device. (2)

[0718] The transmitting device according to (1), wherein:

[0719] The area specifying unit specifies position information about the difference area in the second shelf image, and

[0720] The sending processing unit sends the location information. (2)

[0722] The transmitting device according to (2), wherein the position information is coordinate information in the second shelf image. (4)

[0724] The transmitting device according to (2), wherein the position information is information for specifying a position at which the product included in the difference area is displayed. (5)

[0726] The transmitting device according to any one of (1) to (4), wherein the transmission processing unit compresses the first shelf image and the partial image and transmits the first shelf image and the partial image to the receiving device. (6)

[0728] The transmitting device according to any one of (1) to (5), wherein when it is determined that an obstruction of a product displayed on the product shelf has occurred in the first shelf image, the image acquisition unit acquires the first shelf image again. (7)

[0730] The transmitting device according to any one of (1) to (6), further comprising:

[0731] an imaging unit, capturing a first shelf image and a second shelf image,

[0732] The imaging unit captures the second shelf image at predetermined time intervals. (8)

[0734] The sending device according to (7), further comprising:

[0735] A battery, providing the supply voltage;

[0736] Wherein, the imaging unit changes each predetermined time according to the remaining battery power of the battery. (9)

[0738] A receiving device, comprising:

[0739] a receiving processing unit that receives a partial image of a second shelf image cut out to include a difference area between a first shelf image captured at a first time for a product shelf on which products are displayed and a second shelf image captured at a second time after the first time;

[0740] an image recognition processing unit that performs image recognition processing on a portion of the image to recognize a state of the product; and

[0741] The notification processing unit performs notification processing according to the status of the product. (10)

[0743] The receiving device according to (9), wherein the status of the product is a status related to the quantity of the product. (11)

[0745] The receiving device according to (9), wherein the state of the product is a state related to a displayed posture of the product. (12)

[0747] The receiving device according to (9), wherein the state of the product is a state related to a display position. (13)

[0749] The receiving device according to (9), wherein the status of the product is a status related to a price tag of the product. (14)

[0751] The receiving device according to any one of (9) to (13), wherein:

[0752] capturing a second shelf image at predetermined time intervals,

[0753] The notification processing unit performs notification processing for each product and for each type of notification content, and

[0754] The receiving device further includes a count processing unit that counts the number of times the notification process has been performed for each product and for each type of notification content, and

[0755] The instruction processing unit instructs to change each predetermined time according to the counted number of times. (15)

[0757] The receiving device according to any one of (9) to (14), further comprising:

[0758] A display information generating unit that generates display information that displays the notification content of the notification process and the second shelf image; and

[0759] Send processing unit, send display information. (16)

[0761] The receiving device according to (15), wherein the display information is information in which information on the quantity of displayed products is superimposed on each display area of ​​each product in the second shelf image. (17)

[0763] The receiving device according to (15) or (16), wherein, when the same product is displayed in different display sections in the product shelf, the display information includes information for identifying grouping information for grouping the display sections displaying the same product. (18)

[0765] The receiving device according to any one of (15) to (17), wherein the display information includes history information on the quantity of the displayed product. (19)

[0767] An information processing method for causing a computer device to execute:

[0768] Processing for acquiring a first shelf image captured at a first time for a product shelf having products displayed thereon and a second shelf image captured at a second time for the product shelf after the first time;

[0769] a process for specifying a difference area between a first shelf image and a second shelf image;

[0770] processing for cutting out a portion of the image from the second shelf image so as to include the difference area; and

[0771] The process of sending the first shelf image and the partial image to a receiving device. (20)

[0773] An information processing method for causing a computer device to execute:

[0774] a process for receiving a partial image of a second shelf image cropped to include a difference region between a first shelf image captured at a first time for a product shelf on which products are displayed and a second shelf image captured at a second time subsequent to the first time;

[0775] a process for performing image recognition processing on a portion of the image to recognize the state of the product; and

[0776] Notification processing according to the status of the product.

[0777] Reference Mark List

[0778] 1 Cloud Server

[0779] 3. 3A, 3B cameras

[0780] 38 Power supply unit (battery)

[0781] 41 Imaging Unit

[0782] F1 Image Acquisition Unit

[0783] F3 area designation unit

[0784] F4 Switch out processing unit

[0785] F5 Send Processing Unit

[0786] F11 receiving processing unit

[0787] F13 Image Recognition Processing Unit

[0788] F14 Notification Processing Unit

[0789] F15 counting processing unit

[0790] F16 indicates the processing unit

[0791] F21 Display information generation unit

[0792] F22 Send processing unit.

Claims

1. A sending device, comprising: An image acquisition unit, acquiring a first shelf image captured at a first time for a product shelf on which products are displayed and a second shelf image captured at a second time for the product shelf after the first time; an area specifying unit, which specifies a difference area between the first shelf image and the second shelf image; a cutting-out processing unit, cutting out a partial image from the second shelf image so as to include the different area; as well as The sending processing unit sends the first shelf image and the partial image to a receiving device.

2. The transmitting device according to claim 1, wherein: The area specifying unit specifies position information about the different area in the second shelf image, and The sending processing unit sends the location information.

3. The transmitting device according to claim 2, wherein: The position information is coordinate information in the second shelf image.

4. The transmitting device according to claim 2, wherein: The position information is information for specifying a position at which the product included in the difference area is displayed.

5. The transmitting device according to claim 1, wherein: The transmission processing unit compresses the first shelf image and the partial image and transmits the first shelf image and the partial image to the receiving device.

6. The transmitting device according to claim 1, wherein: When it is determined that occlusion of the product displayed on the product shelf has occurred in the first shelf image, the image acquisition unit acquires the first shelf image again.

7. The transmitting device according to claim 1, further comprising: an imaging unit, capturing the first shelf image and the second shelf image, Wherein, the imaging unit captures the second shelf image at predetermined intervals.

8. The sending device according to claim 7, further comprising: A battery, providing the supply voltage; Wherein, the imaging unit changes the predetermined time according to the remaining battery power of the battery.

9. A receiving device, comprising: a receiving processing unit that receives a partial image of a second shelf image, the second shelf image being cut out to include a difference area between a first shelf image captured at a first time for a product shelf on which products are displayed and the second shelf image captured at a second time after the first time; an image recognition processing unit, performing image recognition processing on the partial image to recognize the state of the product; as well as A notification processing unit performs notification processing according to the status of the product.

10. The receiving device according to claim 9, wherein: The status of the product is a status related to the quantity of the product.

11. The receiving device according to claim 9, wherein: The state of the product is a state related to the posture of the displayed product.

12. The receiving device according to claim 9, wherein: The status of the product is a status related to the display position.

13. The receiving device according to claim 9, wherein: The status of the product is a status associated with a price tag of the product.

14. The receiving device according to claim 9, wherein: capturing the second shelf image at predetermined intervals, The notification processing unit performs the notification processing for each of the products and for each type of notification content, and The receiving device further includes a count processing unit that counts the number of times the notification process has been performed for each of the products and for each type of the notification content, and The instruction processing unit instructs to change the predetermined time according to the counted number of times.

15. The receiving device according to claim 9, further comprising: a display information generating unit, generating display information for displaying the notification content of the notification process and the second shelf image; as well as The sending processing unit sends the display information.

16. The receiving device according to claim 15, wherein: The display information is information in which information on the quantity of displayed products is superimposed on each display area of ​​each product in the second shelf image.

17. The receiving device according to claim 15, wherein: When the same product is displayed in different display parts in the product shelf, the display information includes information for identifying grouping information for grouping the display parts displaying the same product.

18. The receiving device according to claim 15, wherein: The display information includes historical information about the quantity of the displayed product.

19. An information processing method, for causing a computer device to execute: Processing for acquiring a first shelf image captured at a first time for a product shelf having products displayed thereon and a second shelf image captured at a second time for the product shelf after the first time; a process for specifying a difference area between the first shelf image and the second shelf image; processing for cutting out a portion of the image from the second shelf image so as to include the difference area; as well as The process of transmitting the first shelf image and the partial image to a receiving device.

20. An information processing method, for causing a computer device to execute: a process for receiving a partial image of a second shelf image cropped to include a difference region between a first shelf image captured at a first time for a product shelf on which products are displayed and the second shelf image captured at a second time subsequent to the first time; processing for performing image recognition processing on the partial image to identify the status of the product; as well as Notification process according to the status of the product.

Images