Information processing system and information processing method

Abstract

To provide an information processing system capable of effectively coping with a fraudulence such as shoplifting at a shop where customers are dealt with remotely.SOLUTION: A shop (100) is equipped with a customer service monitor (102), a customer service camera (104), a microphone (106), and a loudspeaker (108) in relation to a shot terminal (12). Further, a settlement machine (110) and a monitoring camera (120) are disposed at predetermined positions. Based on camera videos of the customer service camera and monitoring camera, the shop terminal detects as a trigger event a suspicious action of a customer (CUS) which has a possibility of being linked to shoplifting (S3, and S11 to S114). In response to the trigger event, an image (AVT) of an avatar displayed on the customer service monitor is forced to give a warning (S5).SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] This invention relates to an information processing system and an information processing method, and particularly, for example, to an information processing system and an information processing method in a store that remotely receives customers by displaying an operator's avatar.

Background Art

[0002] An example of this type of information processing system is disclosed in Patent Document 1. The communication system disclosed in this Patent Document 1 provides a remote customer service.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, in order to balance labor cost reduction and staffing needs, as well as for security purposes, retail, service, and financial industries have been reducing the number of employees working in stores and shifting some services to remote customer service. However, the underlying technology In a remote customer service such as this, compared to the case of dealing with customers face-to-face in a store, dealing with theft and other irregularities becomes a problem.

[0005] Therefore, the main object of this invention is to provide a novel information processing system and an information processing method.

[0006] Another object of this invention is to provide an information processing system and an information processing method that can effectively respond to irregularities such as theft in a store that remotely receives customers.

Means for Solving the Problems

[0007] To address the above-mentioned issues, this disclosure employs the following configuration. Reference numerals and supplementary explanations in parentheses indicate correspondences with embodiments described to aid understanding of this disclosure and do not limit this disclosure in any way.

[0008] The first embodiment is an information processing system operated in a store where a camera is installed to photograph customers and an operator provides customer service through an avatar displayed on a customer service monitor, comprising: detection means for detecting suspicious customer behavior as a trigger event based on camera footage from the camera, and warning means for executing a warning display on the customer service monitor in response to the trigger event. The warning mechanism involves activating an avatar displayed on the customer service monitor. It is an information processing system.

[0009] In the first embodiment, the information processing system (10: a reference numeral used to illustrate a part corresponding to the embodiment, but not intended to limit it; the same applies hereinafter) is operated in a store (100) equipped with cameras (104, 120) that photograph customers, and where an operator provides customer service through an avatar (AVT) displayed on a customer service monitor (102). The detection means (20, S3, S11-S14) detects suspicious customer behavior, such as shoplifting, as a trigger event based on camera images from the cameras. warning means (20, S5) is triggered by the customer service monitor in response to the trigger event. caveat Execute this. The warning system uses an avatar displayed on a customer service monitor to trigger an alert.

[0010] According to the first embodiment, fraudulent activities such as shoplifting by customers can be effectively prevented. The second embodiment is an information processing device dependent on the first embodiment, wherein the warning means provides a warning by operating the avatar so that the avatar's face, displayed on the customer service monitor, faces the customer. The third embodiment is an information processing device dependent on the first embodiment, wherein the warning means provides a warning by operating the avatar displayed on the customer service monitor so that the avatar's gaze is directed toward the customer. The fourth embodiment is an information processing device dependent on the first embodiment, wherein the warning means issues a warning by making the avatar displayed on the customer service monitor move as if waving its hand. The fifth embodiment is an information processing device dependent on the first embodiment, wherein the warning means provides a warning by having an avatar displayed on a customer service monitor emit a voice.

[0011] The 6 The embodiment is an information processing system dependent on the first embodiment, wherein the detection means detects the customer's facial behavior as a trigger event.

[0012] The 6In the embodiment, the detection means detects, for example, the facial behavior of a customer such as fidgeting.

[0013] No. 7 The embodiment is for information processing in an information processing system that is used in a store where an operator serves customers through an avatar displayed on a customer service monitor, and is provided with a camera for photographing customers. Programs executed by the device's computer That is, The program uses the computer, detecting a suspicious behavior of a customer as a trigger event based on the camera video by the camera, detection means and executing a warning display by the customer service monitor according to the trigger event. The program functions as a warning mechanism, and the warning mechanism works by activating an avatar displayed on the customer service monitor. That's it.

[0014] No. 7 According to the embodiment, the same effects as those of the first embodiment can be expected.

Advantages of the Invention

[0015] According to this invention, it is possible to effectively deal with customer fraud such as shoplifting in a store that provides remote customer service.

[0016] The above object, other objects, features, and advantages of this invention will become clearer from the following detailed description of the embodiments made with reference to the drawings.

Brief Description of the Drawings

[0017] [Figure 1] FIG. 1 is a diagram showing an information processing system according to an embodiment of this invention. [Figure 2] FIG. 2 is a diagram showing an example of a store using the information processing system according to an embodiment of this invention. [Figure 3] FIG. 3 is a block diagram showing an example of the electrical configuration of the store terminal shown in FIG. 2. [Figure 4] FIG. 4 is a block diagram showing an example of the electrical configuration of the operator terminal shown in FIG. 2. [Figure 5] FIG. 5 is a diagram showing an example of the memory map of the RAM of the store terminal shown in FIG. 2. [Figure 6]Figure 6 is a flowchart showing an example of the overall information processing performed by the CPU of the store terminal shown in Figure 3. [Figure 7] Figure 7 is a flowchart showing an example of the trigger event detection process in the overall information processing shown in Figure 6. [Figure 8] Figure 8 is a flowchart showing a specific example of the trigger event detection process shown in Figure 7. [Figure 9] Figure 9 is a flowchart illustrating another specific example of the trigger event detection process shown in Figure 7. [Figure 10] Figure 10 is a flowchart showing yet another specific example of the trigger event detection process shown in Figure 7. [Figure 11] Figure 11 is a flowchart illustrating another specific example of the trigger event detection process shown in Figure 7. [Modes for carrying out the invention]

[0018] Referring to Figure 1, the information processing system 10 of this embodiment includes a store terminal 12, which is communicated to an operator terminal 16 via a network 14.

[0019] For example, using WebRTC (Web Real-Time Communication) technology, the store terminal 12 and the operator terminal 16 can communicate via P2P (Peer to Peer), allowing video and / or audio to be sent and received in real time via a web browser.

[0020] Although not shown in the diagram, WebRTC can be used in combination with multiple servers. Specifically, the multiple servers used for WebRTC include a signaling server, a STUN server, and a TURN server. Although not explained in detail, the signaling server is a server that obtains information about the communication partner via WebRTC. The STUN server and TURN server are servers that perform so-called NAT (Network Address Translation) traversal when the communication partner is on a different network.

[0021] In this embodiment, one store terminal 12 and one operator terminal 16 are shown, but in reality, multiple store terminals 12 and multiple operator terminals 16 are connected to the network 14, and a connection state is established for one of the store terminals 12 and one operator terminal 16 that is configured to conduct video calls or web conferences.

[0022] Furthermore, if the operator terminal 16 to be connected to the store terminal 12 is predetermined, and the store terminal 12 has connection information for the operator terminal 16, and the operator terminal 16 has connection information for the store terminal 12, then the connection status between the store terminal 12 and the operator terminal 16 can be established without using the server described above.

[0023] The store terminal 12 is basically installed in a store 100 (Figure 2) to which the information processing system 10 is applied, for example, in a supermarket, and is used by a customer (CUS) (Figure 2) or an employee of the store. The operator terminal 16 is used by an operator (not shown).

[0024] The store terminal 12 is an information processing device, and is either a notebook PC or a desktop PC, but other devices such as smartphones or tablet PCs can also be used. However, when using a notebook PC, smartphone, or tablet PC, a relatively large display device for displaying avatar images may be prepared separately and connected to the notebook PC, smartphone, or tablet PC.

[0025] Network 14 consists of an IP network (or IP network) including the Internet, and an access network (or access network) for accessing this IP network. The access network can include public telephone networks, mobile phone networks, wired LANs, wireless LANs, CATV (Cable Television), etc.

[0026] The operator terminal 16 is an information processing device different from the store terminal 12. For example, it may be a notebook PC or a desktop PC, but other devices such as a smartphone or tablet PC can also be used.

[0027] Now, referring to Figure 2, we will explain the parts necessary to describe the embodiment in store 100.

[0028] Inside the store 100, a customer service monitor 102 is provided. In this embodiment, the customer service monitor 102 displays an avatar image AVT corresponding to the operator's avatar, and the operator interacts with the customer CUS through this avatar image AVT.

[0029] The operator interacts with the customer user (CUS) by controlling the avatar through the operator terminal 16, the network 14, and the store terminal 12, that is, by controlling the avatar's movements and speech. In other words, the customer user (CUS) is a user of the customer service provided by the operator at the operator terminal 16 using the avatar.

[0030] A customer service camera 104 is provided in a predetermined location related to the customer service monitor 102, for example, on the top edge of the customer service monitor 102. The customer service camera 104 is positioned to capture the face of the customer CUS in front of the customer service monitor 102, i.e., facing the customer service monitor 102. A microphone 106 is provided in the same position as the customer service camera 104 (or a different position). The microphone 106 is positioned to detect the voice of the customer CUS in front of the customer service monitor 102.

[0031] Furthermore, a speaker 108 is provided at the top of the customer service monitor 102, for example. The speaker 108 is positioned to deliver sound to the customer user (CUS) in front of the customer service monitor 102. Therefore, the operator, or avatar, interacts with the customer user (CUS) through the microphone 106 and speaker 108 while providing customer service. For example, the operator can answer questions from the customer user (CUS) or provide information to the customer user (CUS).

[0032] A payment machine 110 is also provided within the store 100. The payment machine 110 is a so-called self-checkout machine and is equipped with a scanner unit 112 for reading barcodes (not shown) attached to products, a payment unit 114 for settling payments with cards (prepaid cards, credit cards, etc.) or cash, and a display 116 for displaying the status of operations such as scanning barcodes with the scanner unit 112 and making payments using the payment unit 114.

[0033] A surveillance camera 120, primarily used to photograph the area around the payment machine 110, is installed, for example, above the payment machine 110. This surveillance camera 120 is used to monitor the payment status of the customer (CUS).

[0034] In such a store 100, an avatar image ATV corresponding to the operator is displayed on the customer service monitor 102, and the avatar's voice is output from the speaker 108. The operator who is responding to the customer (CUS) can view the video footage captured by the customer service camera 104 and the surveillance camera 120, and listen to the customer's voice detected by the microphone 106. In other words, the operator responds to the customer (CUS) by viewing the customer's (face) video and listening to the customer's voice through the avatar.

[0035] The avatar's actions when the operator interacts with a customer (CUS) are controlled by the operator. For example, video data that performs pre-set actions is played. In this case, the operator instructs the avatar to move, and the store terminal 12 plays video data corresponding to the instructed action, and the video is displayed on the customer service monitor 102. In another example, the avatar's action data can be generated based on the operator's actions, and the avatar's actions can be controlled by making it move according to the action data, thereby controlling the avatar's image (AVT) displayed on the customer service monitor 102. The action data can be generated from captured images of the operator using an image processing library such as MediaPipe Holistic. This action data is information (data) about the position and orientation of each part of the avatar to control its gestures, facial orientation, and facial expressions.

[0036] However, if the operator is not attending to a customer, the customer service monitor 102 will display a video or still image of an avatar performing a predetermined action. Alternatively, it may display information about the store or information about the products sold in the store.

[0037] Figure 3 is a block diagram showing an example of the electrical configuration of the store terminal 12 shown in Figure 1. As shown in Figure 3, the store terminal 12 includes a CPU 20, which is connected to RAM 22, a communication interface (communication I / F) 24, and an input / output interface (input / output I / F) 26 via an internal bus.

[0038] The CPU 20 is responsible for the overall control of the store terminal 12. However, instead of the CPU 20, a System-on-a-chip (SoC) that includes multiple functions such as CPU functionality and GPU (Graphics Processing Unit) functionality may be provided.

[0039] RAM22 is the main memory and is used as the work area or buffer area for the CPU20. Although not shown in the diagram, the store terminal 12 is equipped with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD.

[0040] The communication interface 24 is a wired interface for sending and receiving control signals and data to and from an external computer, such as an operator terminal 16, via the network 14 under the control of the CPU 20. However, a wireless interface such as Wi-Fi or Bluetooth® can also be used as the communication interface 24.

[0041] The input / output interface 26 is connected to an input device 28, a customer service monitor 102, a customer service camera 104, a microphone 106, a speaker 108, and a surveillance camera 120.

[0042] The input device 28 includes a keyboard, a computer mouse, and a touch panel. However, the touch panel is provided on the display surface of the customer service monitor 102.

[0043] The customer service monitor 102 is, for example, a liquid crystal display. The microphone 106 is a sound-collecting microphone. The speaker 108 is a stereo speaker. The customer service camera 104 and the surveillance camera 120 are CCD cameras, respectively.

[0044] Furthermore, the input / output interface 26 outputs operation data (or operation information) received from the input device 28 to the CPU 20, and outputs image data generated by the CPU 20 to the customer service monitor 102, displaying a screen or image corresponding to the image data on the customer service monitor 102. However, image data received from an external computer (for example, the operator terminal 16) may also be output by the CPU 20.

[0045] Furthermore, the input / output I / F 26 converts customer voices detected by the microphone 106 into digital audio data (customer voice data) and outputs it to the CPU 20, converts the audio data output by the CPU 20 into an analog audio signal and outputs it from the speaker 108, outputs video data of the footage including customer CUS captured (detected) by the customer service camera 104 to the CPU 20, and outputs video data of the usage status of the payment machine 110 captured (detected) by the surveillance camera 120 to the CPU 20.

[0046] However, in this embodiment, the audio data output from the CPU 20 is either the operator's audio data received from the operator terminal 16 or audio data obtained by converting the operator's audio into the avatar's audio. Hereafter, these audio data will be referred to as operator audio data.

[0047] The payment machine 110 is connected to the CPU 20 via a bus and a payment machine interface (payment machine I / F) 30. The payment machine I / F 30, although not shown in the figure, includes a computer or processor that controls various operations in the payment machine 110. For example, barcode data read by the scanner unit 112 is sent to the CPU 20, stored in the RAM 22, and displayed on the display 116 (Figure 2). Similarly, card data read by the settlement unit 114 is sent to the CPU 20, stored in the RAM 22, and displayed on the display 116.

[0048] Note that the electrical configuration of the store terminal 12 shown in Figure 3 is just one example and is not limited to this configuration.

[0049] For example, although not shown in the diagram, environmental sensors such as cameras other than the aforementioned cameras 104 and 120, and microphones other than microphone 106, may be installed within the store 100.

[0050] Figure 4 is a block diagram showing the electrical configuration of the operator terminal 16 shown in Figure 1. As shown in Figure 4, the operator terminal 16 includes a CPU 32, which is connected to the RAM 34, communication I / F 36, and input / output I / F 38 via an internal bus.

[0051] The CPU 32 is responsible for the overall control of the operator terminal 16. However, instead of the CPU 32, an SoC (System on a Chip) with multiple functions such as CPU and GPU capabilities may be provided.

[0052] RAM34 is the main memory and is used as the work area or buffer area for the CPU32. Although not shown in the diagram, the operator terminal 16 is equipped with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD.

[0053] Communication I / F36 is a wired interface that, under the control of the CPU 32, transmits and receives control signals and video and audio data to and from an external computer such as the operator terminal 16 via the network 14. However, a wireless interface such as Wi-Fi or Bluetooth® can also be used as Communication I / F36.

[0054] The input / output interface 38 is connected to an input device 40, a display device 42, a microphone 44, a speaker 46, and a camera 48.

[0055] The input device 40 is a keyboard and a computer mouse. A touch panel may also be provided. However, if a smartphone or tablet PC is used as the operator terminal 16, the input device 40 is a touch panel and hardware buttons.

[0056] Display device 42 is, for example, a liquid crystal display device. Microphone 44 is a sound-collecting microphone. Speaker 46 is a stereo speaker. Camera 48 is a CCD camera.

[0057] Furthermore, the input / output interface 38 outputs operation data (or operation information) received from the input device 40 to the CPU 32, and also outputs image data generated by the CPU 32 to the display device 42, causing the display device 42 to display a screen corresponding to the image data. However, image data received from an external computer (for example, the store terminal 12) may also be output by the CPU 32. The image data received from the store terminal 12 consists of customer video data and screen video data.

[0058] Furthermore, the input / output interface 38 converts the operator's voice detected by the microphone 44 into digital audio data and outputs it to the CPU 32, converts the audio data output by the CPU 32 into an analog audio signal and outputs it from the speaker 46, and outputs video data of the operator's video captured (detected) by the camera 48 to the CPU 32. However, in this embodiment, the audio data output from the CPU 32 is the audio data of the customer CUS received from the store terminal 12. Also, the operator's video is a moving image.

[0059] Note that the electrical configuration of the operator terminal 16 shown in Figure 3 is just one example and is not limited to this configuration.

[0060] Furthermore, if the operator terminal 16 is a smartphone, it is equipped with a call circuit for making calls via a mobile phone network, or a mobile phone network and a public telephone network. However, in this embodiment, such calls are not made, so they are omitted from the illustration.

[0061] Figure 5 is an illustrative diagram showing an example of the memory map of the RAM 22 of the store terminal 12 shown in Figure 4. As shown in Figure 5, the RAM 22 includes a program storage area 50 and a data storage area 52. The control program of the store terminal 12 is stored in the program storage area 50.

[0062] The control program executed on the store terminal 12 includes a main processing program 54a, an operation detection program 54b, a communication program 54c, a camera image acquisition program 54d, an audio acquisition program 54e, a trigger event detection program 54f, a warning program 54g, and the like.

[0063] The main processing program 54a is a program for executing the main routine of the control processing of the store terminal 12 in this embodiment. This main processing program 54a also includes a program that executes processing related to the payment machine 110 via the payment machine I / F 30, but this program is not directly related to the embodiment of this disclosure, so its description is omitted.

[0064] The operation detection program 54b is a program that detects operation data input from the input device 28 and stores it in the operation data storage area 56a (described later) of the data storage area 52.

[0065] The communication program 54c is a program for communicating with external devices, such as the operator terminal 16.

[0066] The camera video acquisition program 54d is a program for acquiring images of customers (CUS) and the inside of the store 100 that are being captured by cameras such as the customer service camera 104 and the surveillance camera 120, which are installed in conjunction with the store terminal 12.

[0067] Such camera images are stored as video data in the camera video data storage area 56c (described later) of the data storage area 52, and are transmitted to the operator terminal 16 by the communication program 54c as needed.

[0068] The voice acquisition program 54e is a program for acquiring customer CUS and voices from within the store 100, which are picked up by microphones such as microphone 106 that are provided in conjunction with the store terminal 12.

[0069] The voice acquired in this manner is stored as voice data in the voice data storage area 56d (described later) of the data storage area 52, and is transmitted to the operator terminal 16 by the communication program 54c as needed.

[0070] The trigger event detection program 54f is designed to detect suspicious behavior by a customer (CUS) that may lead to shoplifting, as an event that triggers the warning program 54g described later.

[0071] In this embodiment, four types of trigger events are assumed, including, for example, face behavior detection, gaze detection, barcode motion detection, and gaze point detection.

[0072] The facial behavior detection program 54f1 is a program that detects suspicious facial behavior of the customer CUS, such as "looking around nervously," based on camera video data 56c.

[0073] The gaze detection program 54f2 is a program that detects, based on camera video data 56c, when the customer's gaze is directed, for example, towards the customer service monitor 102.

[0074] The barcode operation detection program 54f3 is a program that detects, for example, whether or not a customer CUS is performing a barcode operation near the payment machine 110, based on camera video data 56c.

[0075] The gaze detection program 54f4 is a program that detects, based on camera video data 56c, that a customer's gaze is fixed on a specific location, such as the location of the surveillance camera 120.

[0076] The warning program 54g is a program that, when a trigger event is detected by the trigger event detection program 54f, executes a warning display that causes the face of the avatar image AVT displayed on the customer service monitor 102 to be intentionally turned towards the customer CUS, or causes the avatar image AVT to direct its gaze towards the customer CUS, or executes a warning display that shows a text overlay on the customer service monitor 102 (for example, a text overlay that says "Security camera in operation!").

[0077] However, the specific form of the warning is not limited to this, and any other warning is possible, such as performing other warning displays using the customer service monitor 102, or outputting a warning sound (voice) from the speaker 108 instead of a warning display, or in conjunction with a warning display.

[0078] The warnings (by image and / or sound) executed by this warning program 54g can also be configured to execute warnings that correspond individually to events detected by each of the trigger event detection programs 54f1-54f4 of the aforementioned trigger event detection program 54f. To do this, the trigger event detection flag 56f, described later, should be used to individually store the occurrence of events detected by each of the programs 54f1-54f4.

[0079] Although not shown in the diagram, the program storage area 50 stores the operating system and middleware of the store terminal 12, as well as a browser and other application programs other than the control program disclosed herein. However, since these other application programs are separate programs from the control program, i.e., the information processing program, in this embodiment, they are not included in the information processing program.

[0080] The RAM22's data storage area 52 stores operation data storage area 56a, warning data storage area 56b, camera image data storage area 56c, audio data storage area 56d, system setting data storage area 56e, and trigger event detection flag storage area 56f, among others.

[0081] The operation data storage area 56a temporarily stores operation data for the store terminal 12 detected according to the operation detection program 54b. The operation data is used for processing by the CPU 20 and transmitted to the operator terminal 16 as needed.

[0082] The warning data storage area 56b stores, as described above, control data for controlling the posture (orientation) of the avatar image AVT displayed on the customer service monitor 102 when suspicious behavior from a customer CUS is detected, display data for the text displayed on the customer service monitor 102, and audio data for the warning sound output from the speaker 108.

[0083] The camera image data storage area 56c stores the camera image data acquired by the camera image acquisition program 54d.

[0084] The audio data storage area 56d stores the audio data acquired by the audio acquisition program 54e.

[0085] The system configuration data storage area 56e is pre-configured with the necessary settings data for operating the information processing system of this embodiment within the store 100.

[0086] For example, in the gaze detection program 54f2, data indicating the 3D position of each camera in the store (including all cameras such as the customer service camera 104 and the surveillance camera 120) and data indicating the normal direction of the customer service monitor 102 are pre-stored so that the human body of a customer CUS can be detected based on camera video data.

[0087] Furthermore, the barcode operation detection program 54f3 pre-stores location data indicating the area within the store where barcode processing is performed, for example, within the shooting range of the surveillance camera 120, that is set as the area where the customer CUS will perform barcode operations. This area can be called the barcode processing area where barcode operations are expected to occur.

[0088] The trigger event detection flag storage area 56f stores, for example, a 4-bit trigger event detection flag, which stores the four types of trigger events detected by the trigger event detection program 54f described above in their respective corresponding bits. For example, the trigger event detection flag stores the event detected by the face behavior detection program 54f1 (suspicious facial behavior of the customer) in the first bit, the event detected by the gaze detection program 54f2 (the customer's gaze was directed towards the customer service monitor 102) in the second bit, the event detected by the barcode operation detection program 54f3 (no barcode operation) in the third bit, and the event detected by the gaze detection program 54f4 (the customer gazed at a specific location) in the last bit.

[0089] However, if the warning program 54g issues the same warning regardless of the type of trigger event detected, the trigger event detection flag can be a 1-bit flag.

[0090] Although not shown in the diagram, the data storage area 52 stores other data necessary for the execution and operation of the control program of the store terminal 12, as well as flags and other counters (timers) necessary for the execution of the control program.

[0091] The operation of the information processing system 10 in this embodiment will now be described. In this information processing system 10, the operator basically interacts with customers (CUS) via an avatar image (AVT) displayed on a customer service monitor (AVT) in the store (Store 100) from the operator terminal (Operator Terminal) (16) when necessary. However, customer service itself is not a crucial part of this embodiment, so its explanation will be omitted.

[0092] Referring to Figure 6, the CPU 20 of the store terminal 12 constantly monitors whether a trigger event is detected in step S1. The specific method for detecting a trigger event will be explained later with reference to Figures 7-11.

[0093] Please note that each processing operation shown in the flowcharts of Figures 6-11 is executed repeatedly, for example, every frame.

[0094] Then, in step S3, the CPU 20 determines whether a trigger event has been detected. In this step S3, the CPU 20 can determine whether a trigger event has been detected by referring to the trigger event detection flag 56f in the data storage area 52.

[0095] If "YES" is determined in step S3, that is, if some trigger event is detected, a warning is issued in step S5.

[0096] A concrete example of a warning is the movement of the avatar's image AVT displayed on the customer service monitor 102 to face the direction of the customer CUS. In this warning, the avatar's movement involves the CPU 20 first analyzing camera footage from, for example, the customer service camera 104, the surveillance camera 120, or other cameras within the store to detect the customer CUS's position. Then, the CPU 20 controls the display of the avatar's image AVT so that the avatar faces that direction, according to warning data stored, for example, in the warning data storage area 56b. However, the display of the avatar's image AVT may also be controlled so that the avatar's gaze is directed towards the customer CUS. If the Mona Lisa effect is to be considered, the avatar's gaze may be moved by a predetermined angle (for example, about 5 to 10 degrees) in the direction of the customer CUS as seen from the avatar (customer service monitor 102).

[0097] Another example of a warning is to display a text overlay on the customer service monitor 102, such as "Security cameras are in operation." In this method, the CPU 20 reads the text overlay display data contained in the warning data stored in the warning data storage area 56b, for example, and displays it on the customer service monitor 102.

[0098] Another example of a warning is outputting a warning sound from speaker 108. In this method, the CPU 20 reads the warning sound data contained in the warning data stored in the warning data storage area 56b, for example, and outputs it from speaker 108. However, the warning sound does not have to be a specific voice message such as "Security cameras are in operation," but rather simple voice data such as a buzzer sound or a bell sound.

[0099] According to this embodiment, for example, when suspicious behavior from a customer (CUS) that suggests shoplifting is detected as a trigger event, a warning is issued, thus enabling an effective response to fraudulent activities such as shoplifting by a customer (CUS).

[0100] As shown in Figure 7, in this embodiment, four types of trigger events are set, which are detected by the face behavior detection program 54f1, gaze detection program 54f2, barcode motion detection program 54f3, and gaze detection program 54f4 shown in Figure 5, respectively.

[0101] Referring to Figure 8, the face behavior detection process begins when the customer CUS's face is detected. For example, when the customer CUS's face is detected based on camera video data from the customer service camera 104 (Figure 2), in the first step S111, the CPU 20 estimates the normal vector of that face (a vector perpendicular to the face's surface). However, since a face is not actually a plane, it is difficult to strictly define it as a normal vector, but for convenience, the front of the face is treated as a plane and the normal vector is estimated.

[0102] In the following step S112, the CPU 20 monitors the amount of rotation of the normal vector. In other words, it detects how much the normal vector has rotated relative to the normal vector detected in step S111.

[0103] Then, in step S113, the CPU 20 determines whether the amount of rotation of the normal vector is greater than or equal to a predetermined angle, for example, 30 degrees.

[0104] If CPU20 determines "NO" in step S113, it terminates this subroutine.

[0105] If the CPU 20 determines "YES" in step S113, in step S114, the CPU 20 increments (+1) the count counter (not shown) provided in the data storage area 52.

[0106] In the next step S115, the CPU 20 determines whether the count value of the counter has reached a predetermined number of times, for example, 10 times. Here, if the count value has reached a predetermined number of times, it means that the customer CUS has shaken their head from side to side several times, and therefore the CPU 20 determines that the customer CUS has looked around.

[0107] If the determination in step S115 is "NO", this subroutine terminates and the system returns to the main routine in Figure 6. If the determination in step S115 is "YES", the CPU 20 determines that it has detected facial behavior, sets the corresponding bit (for example, the leading bit) of the trigger event detection flag in the data storage area 54 to "1", terminates this subroutine, and returns to the main routine. Therefore, in step S5 of Figure 6, the warning processing described earlier is executed.

[0108] Next, referring to Figure 9, the process of detecting eye movements on the monitor will be explained. This subroutine also starts when the customer's face is detected.

[0109] As explained earlier, the system setting data storage area 56e is pre-stored with initial settings including data indicating the 3D position of each camera in the store and data indicating the normal direction of the customer service monitor 102. Based on this setting data, the CPU 20 can detect the human body of a customer CUS based on the camera video data, in accordance with the gaze detection program 54f2.

[0110] In the first step S121, the direction (orientation) of the customer CUS's face, which has been detected as a human body, is detected. The CPU 20 uses image processing technology, such as Media Pipe (product name), to estimate the direction of the customer CUS's face relative to the camera that captured the customer CUS, such as the customer service camera 104.

[0111] In the following step S122, the CPU 20 estimates the angle of the customer's face relative to the customer monitor 102 from the angles of the camera that captured the camera footage in which the customer's body was detected, for example, the customer service camera 104 and the customer service monitor 102. Here, for example, the direction of the customer's face as seen from the customer service camera 104 is detected, and the angle of that direction relative to the pre-set normal direction of the customer service monitor 102 can be estimated.

[0112] In the next step S123, the CPU 20 determines whether the angle of the direction of the customer CUS's face, estimated in step S122, relative to the customer service monitor 102 is within a predetermined angle, for example, 60 degrees.

[0113] If the result in step S123 is "NO", the subroutine terminates. However, if the result is "YES", the CPU 20 increments (+1) a count counter (not shown) located in the data storage area 52 in step S124.

[0114] In the next step, S125, the CPU 20 determines whether the count value of the counter has reached a predetermined number of times, for example, 5 times. Here, if the count value has reached the predetermined number of times, it is determined that the customer CUS is continuing to look at the customer service monitor 102, that is, the CPU 20 determines "YES" in step S125, detects the gaze toward the customer service monitor 102 in step S126, and terminates this subroutine. In step S126, the CPU 20 sets, for example, the second bit of the trigger event detection flag to "1". However, if the CPU 20 determines "NO" in step S125, it terminates this subroutine.

[0115] Thus, in a situation where the customer CUS is facing the customer service monitor 102, a possible warning in step S5 of Figure 6 could be, for example, to orient the face of the avatar image AVT displayed on the customer service monitor 102 toward the face of the customer CUS. In that case, the avatar image AVT may also be made to smile or wave, in addition to orienting its face toward that direction. However, these additional actions may be executed based on warning data in storage area 56b of data storage area 54.

[0116] Referring to Figure 10, the barcode operation detection process will be explained. This barcode operation detection process detects whether or not a barcode operation is performed in the barcode processing area, which is set as the initial setting in the system setting data storage area 56e as described earlier.

[0117] In the first step S131, the CPU 20 monitors whether an item has leaked out of the scanner unit 112 of the payment machine 110 in the barcode processing area, based on the camera footage from the surveillance camera 120.

[0118] Based on the monitoring results from step S131, step S132 determines whether or not there has been any leakage of goods.

[0119] If the result in step S132 is "NO", this subroutine terminates. On the other hand, if the result in step S132 is "YES", that is, if it is determined that an item has been released from the area of ​​the scanner unit 112, then in the following step S133, the CPU 20 determines whether a barcode operation occurred. For example, it determines whether there was any movement of an item in front of the scanner unit 112 that would scan a barcode.

[0120] However, when using the handheld scanner of the scanner unit 112, if the handheld scanner is fixed, it is sufficient to detect whether there is movement of a similar item, and if the handheld scanner is moved, it is sufficient to detect the movement of the handheld scanner.

[0121] If the result in step S133 is "YES", then this subroutine is terminated, indicating that a legitimate purchasing action has been taken.

[0122] Conversely, if step S133 determines "NO," that is, if there was no barcode operation despite the goods being released, it is determined that no legitimate purchase action was taken, and in the next step S134, the absence of barcode operation is detected and this subroutine terminates. In step S134, the CPU 20 sets, for example, the third bit of the trigger event detection flag to "1".

[0123] Therefore, the warning process described earlier in step S5 of Figure 6 is executed.

[0124] Next, the gaze detection process shown in Figure 11 starts when the customer CUS's face is detected. For example, when the customer CUS's face is detected based on the camera video data from the customer service camera 104 (Figure 2), in the first step S141, the CPU 20 estimates the normal vector of that face. However, similar to step S111 of the face behavior detection process in Figure 8, the front of the customer CUS's face is conveniently treated as a plane to estimate the normal vector.

[0125] In the following step S142, the CPU 20 monitors the amount of rotation of the normal vector. In other words, it detects how much the normal vector has rotated relative to the normal vector detected in step S141.

[0126] In step S143, the CPU 20 determines whether the normal vector of the customer CUS's face, whose rotation amount was monitored in step S142, is pointing in the direction of the surveillance camera 120. For example, if it is within a predetermined angle, for example, ±20 degrees, with respect to the line connecting the surveillance camera 120 and the customer CUS's face, it can be estimated that the customer CUS is looking in the direction of the surveillance camera 120.

[0127] If "NO" is determined in step S143, this subroutine terminates. On the other hand, if "YES" is determined in step S143, the CPU 20 detects a trigger event that customer CUS is looking at a designated location, such as the surveillance camera 120, and sets the corresponding bit (for example, the last bit) of the trigger event detection flag to "1".

[0128] Therefore, in step S5 (Figure 6), the CPU 20 executes a warning process using the customer service monitor 102, speaker 108, etc.

[0129] As described above, in the embodiment of this disclosure, in the information processing system 10 used in the store 100 that provides remote customer service, the store terminal 12 detects suspicious behavior by a customer (CUS) that may lead to shoplifting, and triggers a warning to the customer (CUS). Therefore, it is expected that shoplifting by customers (CUS) can be effectively prevented.

[0130] However, similar processing can also be performed on the operator terminal 16. When performing this on the operator terminal 16, the program and data shown in Figure 5 are pre-configured in the RAM 34 of the operator terminal 16, and the camera video data and audio data necessary for trigger event detection performed in step S3 of Figure 6 are sent from the store terminal 12 to the CPU 32 (Figure 4) via the communication I / F 24, network 14, and communication I / F 36.

[0131] In the above embodiment, the store terminal includes a payment machine, but a separate store terminal and payment machine may be provided for performing user interaction processing and executing the control processing of this invention.

[0132] Furthermore, while the above embodiment uses four types of trigger events, it is not limited to these. The trigger events may be one or more of the four types, or additional trigger events may be added.

[0133] Furthermore, if the same result can be obtained from each step in the flowchart shown in the above embodiment, the order in which the processes are performed can be changed.

[0134] Furthermore, the various screens and specific numerical values ​​mentioned in the above-described examples are merely illustrative and can be modified as needed. [Explanation of Symbols]

[0135] 10. Information Processing Systems 12 ... Store terminal 14 ... Network 16 ... Operator terminal 20, 34 ...CPU 22, 34…memory 24, 36 ...Communication I / F 100 stores 102... Customer service monitor 104... Customer service camera 106... Mike 108...Speaker 110…Payment machine 120... Surveillance cameras

Claims

1. An information processing system operated in a store where a camera is installed to photograph customers, and an operator interacts with customers through an avatar image displayed on a customer service monitor, A detection means for detecting suspicious customer behavior as a trigger event based on camera images from the aforementioned camera, and The system includes a warning means that executes a warning display via the customer service monitor in response to the trigger event, The warning means is an information processing system that issues a warning by activating the avatar displayed on the customer service monitor.

2. The information processing apparatus according to claim 1, wherein the warning means provides a warning by operating the avatar so that the face of the avatar displayed on the customer service monitor faces the customer.

3. The information processing apparatus according to claim 1, wherein the warning means provides a warning by operating the avatar so that the avatar's gaze, which is displayed on the customer service monitor, is directed toward the customer.

4. The information processing apparatus according to claim 1, wherein the warning means provides a warning by making the avatar displayed on the customer service monitor wave its hand.

5. The information processing apparatus according to claim 1, wherein the warning means provides a warning by having the avatar displayed on the customer service monitor emit a sound.

6. The information processing system according to claim 1, wherein the detection means detects the facial behavior of a customer as a trigger event.

7. A program executed by a computer in an information processing device within an information processing system operated in a store where a camera is installed to photograph customers and an operator interacts with customers through an avatar displayed on a customer service monitor, The aforementioned program computer, A detection means for detecting suspicious customer behavior as a trigger event based on camera images from the aforementioned camera, and A program that functions as a warning means to execute a warning display on the customer service monitor in response to the trigger event, and the warning means issues a warning by activating the avatar displayed on the customer service monitor.

Images