Baggage monitoring device, baggage monitoring method and recording medium
The luggage monitoring device addresses the delay in detecting luggage mix-ups by using image analysis and posture estimation to identify registered owner combinations, facilitating early detection and prevention.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- NEC CORP
- Filing Date
- 2024-12-07
- Publication Date
- 2026-06-11
AI Technical Summary
Existing luggage monitoring systems fail to detect luggage mix-ups early, as monitoring is not initiated until the distance between luggage and the owner exceeds a threshold, leading to delayed detection.
A luggage monitoring device that includes image acquisition, detection, posture estimation, and determination of the person's actions based on positional relationships and posture, with notification actions for unregistered combinations, enabling early detection of luggage mix-ups.
Enables early detection and prevention of luggage mix-ups by accurately identifying registered owner combinations and prompting appropriate actions.
Smart Images
Figure JP2024043326_11062026_PF_FP_ABST
Abstract
Description
Luggage Monitoring Device, Luggage Monitoring Method, and Recording Medium
[0001] The present invention relates to a luggage monitoring device, a luggage monitoring method, and a recording medium.
[0002] In trains, buses, etc., there are known ones in which a dedicated space is provided for placing large luggage such as suitcases, carry-on bags, and baby strollers (hereinafter, these are collectively referred to as "luggage"). In such luggage storage areas, luggage mix-ups often occur.
[0003] Patent Document 1 discloses a forgotten item notification control device that can appropriately record the situation and appropriately notify when there is a forgotten item. The forgotten item notification control device described in the same document links the luggage and the owner when it is determined that the luggage is placed on the wire rack based on the data of a sensor that detects that the owner has placed the luggage in the vehicle and the video data of an omnidirectional camera. Then, based on the relative positional relationship between the luggage and the owner, when the distance between the luggage and the owner is equal to or greater than a threshold distance while the position of the luggage does not change, it is determined that it is a forgotten item. When it is determined that it is a forgotten item, the forgotten item notification control device saves the captured video data in a state where overwriting is prohibited and notifies that it is a forgotten item by displaying it on a display device or the like.
[0004] Japanese Unexamined Patent Application Publication No. 2021 - 132289
[0005] In the forgotten item notification control device of Patent Document 1, there is a problem that the above-mentioned luggage mix-up cannot be detected early because monitoring is not started unless the distance between the luggage and the owner becomes equal to or greater than the threshold distance.
[0006] An object of the present disclosure is to provide a luggage monitoring device, a luggage monitoring method, and a recording medium that can contribute to the early detection of luggage mix-ups in a vehicle.
[0007] From a first perspective, a luggage monitoring device is provided, comprising: acquisition means capable of acquiring images from a camera capable of photographing the luggage storage area inside a vehicle; detection means for detecting a person touching luggage captured in the camera's image; estimation means for estimating the posture of the person touching the luggage; first determination means for determining the person's actions based on the positional relationship between the luggage and the person and the person's posture; second determination means for determining whether the combination of the person and the luggage is a pre-registered combination if it is determined that the person is attempting to take the luggage; and notification means for performing a predetermined notification action if it is determined that the combination of the person and the luggage is not a pre-registered combination.
[0008] From a second perspective, a luggage monitoring method is provided which involves acquiring images from a camera capable of photographing the luggage storage area inside the vehicle, detecting a person touching luggage in the camera's images, estimating the posture of the person touching the luggage, determining the person's actions based on the positional relationship between the luggage and the person and the person's posture, determining whether the person is attempting to take the luggage if it is determined that the combination of the person and the luggage is a pre-registered combination, and performing a predetermined notification action if it is determined that the combination of the person and the luggage is not a pre-registered combination.
[0009] From a third perspective, a recording medium is provided that contains a program that causes a computer to execute the following: a process of acquiring an image from a camera capable of photographing the luggage compartment inside a vehicle; a process of detecting a person touching luggage in the image from the camera; a process of estimating the posture of the person touching the luggage; a process of determining the person's actions based on the positional relationship between the luggage and the person and the person's posture; a process of determining whether the combination of the person and the luggage is a pre-registered combination if it is determined that the person is trying to take the luggage; and a process of performing a predetermined notification action if it is determined that the combination of the person and the luggage is not a pre-registered combination.
[0010] This disclosure makes it possible to provide a luggage monitoring device, a luggage monitoring method, and a recording medium that can contribute to the early detection of luggage mix-ups inside a vehicle.
[0011] This is a diagram showing one configuration of the present disclosure. This is a flowchart illustrating the operation of the present disclosure. This is a diagram illustrating the operation of the present disclosure. This is a diagram showing one configuration of the present disclosure. This is a diagram illustrating the package management information held by the package monitoring device of the present disclosure. This is a diagram illustrating the operation of the present disclosure. This is a diagram showing another configuration of the present disclosure. This is a diagram illustrating another example of the package management information held by the package monitoring device of the present disclosure. This is another diagram illustrating the operation of the present disclosure. This is a diagram showing another configuration of the present disclosure. This is another diagram illustrating the operation of the present disclosure. This is a diagram showing the configuration of the computers constituting the package monitoring device of the present disclosure.
[0012] First, an overview of one embodiment of this disclosure will be described with reference to the drawings. In this disclosure, the drawings are associated with one or more embodiments. The reference numerals in the drawings appended to this overview are provided for convenience as examples to aid understanding and are not intended to limit this disclosure to the illustrated embodiments. In addition, the connecting lines between blocks in the drawings and other references referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows schematically indicate the flow of the main signal (data) and do not exclude bidirectionality. The program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and a display device as needed. This computer device is also configured to communicate with devices (including computers) inside or outside the device via the communication interface, whether wired or wireless. In addition, there are ports or interfaces at the input / output connection points of each block in the figures, but these are omitted from the illustration.
[0013] In one embodiment, the present disclosure can be realized by a luggage monitoring device 10, as shown in Figure 1, comprising an acquisition means 11, a detection means 12, an estimation means 13, a first determination means 14, a second determination means 15, and a notification means 16. More specifically, the acquisition means 11 is capable of acquiring images from a camera C capable of photographing the luggage storage area inside a vehicle. The detection means 12 detects a person touching luggage as captured in the image from the camera C. The estimation means 13 estimates the posture of the person touching the luggage. The first determination means 14 determines the person's actions based on the positional relationship between the luggage and the person, and the person's posture. If the second determination means 15 determines that the person is attempting to take the luggage, it determines whether the combination of the person and the luggage is a pre-registered combination. If the notification means 16 determines that the combination of the person and the luggage is not a combination stored in the storage means, it performs a predetermined notification action.
[0014] The luggage monitoring device 10 configured as described above operates as follows. First, the luggage monitoring device 10 acquires an image from a camera capable of photographing the luggage storage area inside the vehicle (step S01 in Figure 2).
[0015] Next, the luggage monitoring device 10 detects a person touching the luggage as captured in the camera's image (step S02 in Figure 2). If no person touching the luggage is detected, the subsequent processing is omitted.
[0016] Next, the luggage monitoring device 10 estimates the posture of the person who touched the luggage in the luggage storage area (step S03 in Figure 2).
[0017] Next, the luggage monitoring device 10 determines the action the person is about to take based on the relative positions of the luggage and the person, and the person's posture (step S04 in Figure 2). If it determines that the action the person is about to take is not to take the luggage (No. in step S05 in Figure 2), the luggage monitoring device 10 returns to step S01 and continues monitoring. In this way, the determination is made not only based on whether the person touches the luggage, but also using the person's posture, thus preventing false determinations.
[0018] On the other hand, if the determination results in the determination that the person is attempting to take the luggage (Yes in step S05 of Figure 2), the luggage monitoring device 10 determines whether the combination of the person and the luggage is a pre-registered combination (step S06 of Figure 2). If the combination of the person and the luggage is a pre-registered combination, the luggage monitoring device 10 does nothing (permits the removal of the luggage).
[0019] On the other hand, if the device determines that the combination of the person and the luggage is not one of the combinations registered in advance (No. in step S06 of Figure 2), the luggage monitoring device 10 performs a predetermined notification action (step S07 of Figure 2). Examples of such predetermined notification actions include outputting a warning sound or message, notifying the vehicle manager, and notifying the person registered as the owner of the luggage. For example, as shown in Figure 3, the luggage monitoring device 10 outputs the message "Luggage mistake occurred!" on the monitoring screen of the luggage storage area inside the vehicle.
[0020] As explained above, this disclosure makes it possible to detect and address the occurrence of luggage mix-ups inside a vehicle at an early stage.
[0021] [First Embodiment] Next, a first embodiment equipped with a function for registering luggage and its owner when boarding a vehicle will be described. Figure 4 is a diagram showing one configuration of the present disclosure. Referring to Figure 4, a configuration is shown in which a luggage monitoring device 100 connected to a camera C capable of photographing the luggage storage area of the vehicle and a passenger's mobile terminal 200 such as a smartphone can be connected via a network N. Note that the network N may be a cellular network such as LTE (Long Term Evolution) or 5G (Fifth Generation Mobile Communication System), or it may be short-range wireless communication.
[0022] The luggage monitoring device 100 includes an acquisition unit 101, a detection unit 102, an estimation unit 103, a first determination unit 104, a second determination unit 105, a notification unit 106, a management unit 107, and a reception unit 108. Such a luggage monitoring device 100 may be installed in a vehicle, or it may be installed in a control center or the like that can receive images from the vehicle's camera C and send notifications to the vehicle.
[0023] The reception unit 108 provides a registration form for luggage management to the mobile terminal 200, such as a smartphone, of passengers who are scheduled to board, and accepts registration of luggage information, including images of the passenger and their luggage. Specifically, passengers access the luggage monitoring device 100 using their mobile terminal 200, such as a smartphone, at the time of booking or before boarding the vehicle. The passenger then takes pictures of themselves and their luggage using the camera of their mobile terminal 200, following the instructions from the luggage monitoring device 100. The passenger then sends the captured images, etc., to the luggage monitoring device 100 via the registration form. The reception unit 108 sends the received images, etc., of the luggage as management information to the management unit 107. This reception unit 108 corresponds to a reception means that accepts registration of management information, including images of the passenger and their luggage, from passengers at a predetermined time before boarding.
[0024] The management unit 107 retains luggage management information, including images of passengers and luggage, for a certain period of time. Figure 5 shows an example of luggage management information held by the luggage monitoring device 100. In the example in Figure 5, images of luggage received from passengers and images of passengers (boarders) are registered. Note that, as shown in the entries for boarders B1 and B2 in Figure 5, the reception unit 108 may accept registration of images of multiple passengers for a single piece of luggage. This prevents the system from misidentifying luggage when family members or companions take the luggage. The management unit 107 corresponds to a management means that links the passenger's image with the luggage image when the passenger places their luggage in the luggage area. It is desirable that the data held in the management unit 107 be deleted after a certain period of time. In addition, although the luggage monitoring device 100 is equipped with the management unit 107 in the example in Figure 4, the management unit 107 may be located on a network accessible by the luggage monitoring device 100.
[0025] The acquisition unit 101 acquires an image from a camera C capable of photographing the vehicle's luggage area and sends it to the detection unit 102. The detection unit 102 detects a person touching luggage as captured in the image from the camera C. The detection unit 102 can use various image processing engines capable of extracting the area containing the person from the image, or a large-scale language model (LLM) or VLM (Vision-Language Model) capable of performing image analysis. The acquisition unit 101 and the detection unit 102 correspond to the acquisition means 11 and detection means 12 described above, respectively.
[0026] The estimation unit 103 estimates the posture of the detected person. More specifically, the estimation unit 103 performs the following processing. First, the estimation unit 103 obtains the skeletal points of the person detected by the detection unit 102. Next, the estimation unit 103 calculates the characteristic quantities of the person's posture from the coordinates of the person's skeletal points. This estimation unit 103 corresponds to the estimation means 13 described above.
[0027] The first determination unit 104 determines the actions of the person based on the positional relationship between the luggage and the person, and the characteristic quantities of the person's posture. If the determination determines that the person is trying to take the luggage, the first determination unit 104 requests the second determination unit 105 to verify the owner. This first determination unit 104 corresponds to the first determination means 14 described above.
[0028] The first determination unit 104 can determine a person's actions by comparing the positional relationship between the luggage and the person, with the similarity between the posture features of the person extracted from the image and the posture features of pre-prepared model data of a person attempting to take out luggage. By performing determination using such features, accurate determination can be achieved even if the posture, the orientation of the person or luggage differs. Note that the model data does not necessarily have to be obtained from the person (passenger). For example, model data may be created from a large number of sample data of a person attempting to take out luggage, by performing appropriate statistical processing.
[0029] When the second determination unit 105 receives a verification request from the first determination unit 104, it refers to the management information held in the management unit 107 to determine whether the combination of the person and the package is a combination that has been registered in advance. If the determination determines that the combination of the person and the package is not a combination that has been registered in advance, the second determination unit 105 requests the notification unit 106 to perform notification processing. This second determination unit 105 corresponds to the second determination means 15 described above.
[0030] When the notification unit 106 receives a request for notification processing from the second determination unit 105, it performs a predetermined notification operation. This notification unit 106 corresponds to the notification means 16 described above.
[0031] Next, the operation of this embodiment will be described in detail with reference to the drawings. Figure 6 is a diagram illustrating the operation of the present disclosure. First, the luggage monitoring device 100 acquires an image from the camera C (step S101).
[0032] Next, the luggage monitoring device 100 detects a person P touching luggage B, which is captured in the image of camera C (step S102).
[0033] Next, the luggage monitoring device 100 estimates the posture of the detected person P (step S103). More specifically, the luggage monitoring device 100 acquires the skeletal points of the person detected in step S102. Then, the luggage monitoring device 100 calculates the characteristic features of the person's posture from the coordinates of the person's skeletal points. This makes it possible to understand the person's posture in a way that is less affected by the orientation of the camera C, etc.
[0034] Next, the luggage monitoring device 100 determines whether or not the person is attempting to take the luggage, based on the positional relationship between the luggage B and the person P, and the characteristic features of the person P's posture (step S104).
[0035] If the determination results in the above determination that the person is attempting to take the luggage, the luggage monitoring device 100 refers to the management information to determine whether the person P is the owner of luggage B or their companion (step S105). If it is determined that the person P is the owner of luggage B or their companion, no action is taken.
[0036] On the other hand, if the device determines that person P is neither the owner of luggage B nor one of their companions, the luggage monitoring device 100 performs a predetermined notification action (step S106). For example, the luggage monitoring device 100 notifies the vehicle manager, as shown in the speech bubble in the lower right of Figure 6. Upon receiving the notification, the vehicle manager takes action such as rushing to the scene or beginning to monitor the person.
[0037] As explained above, this disclosure makes it possible to detect the occurrence of luggage mix-ups inside a vehicle at an early stage and to prompt appropriate action.
[0038] [Second Embodiment] Next, a second embodiment will be described in which a luggage storage space reservation function and a luggage placement error detection function are added to the luggage monitoring device 100. Figure 7 is a functional block diagram showing another configuration of the luggage monitoring device 100a of this disclosure. The first difference from the luggage monitoring device 100 of the first embodiment shown in Figure 4 is that a luggage storage space reservation function has been added to the luggage monitoring device 100a. The second difference from the luggage monitoring device 100 of the first embodiment is that a function has been added to the luggage monitoring device 100a to detect luggage placement errors (misplacement) in reserved luggage spaces. The other configurations are almost the same as in the first embodiment, so the differences in operation will be explained below.
[0039] The reception unit 108a manages the status of luggage spaces in the luggage storage area (occupied, reserved, available, etc.) and has a function to allocate an available luggage space in the luggage storage area when it receives registration of luggage management information from a passenger, including the passenger's image and images of the luggage.
[0040] The management unit 107a is capable of storing information about the luggage space allocated to the passenger, in addition to the passenger's image and luggage image. Figure 8 shows another example of luggage management information stored by the luggage monitoring device 100a. In the example of Figure 8, in addition to the management information of Figure 5, a reservation location field is added, and information about the luggage space allocated to each passenger is stored. For example, the luggage space allocated to passenger A1 is E2 in front of car 1. This indicates that the luggage space allocated to passenger A1 is section E2 at the front of car 1.
[0041] The first determination unit 104a determines whether a person is taking the action of placing down a luggage, based on the person's action of taking down a luggage, the positional relationship between the luggage and the person, and the characteristic quantities of the person's posture. If it determines that the person is taking the action of placing down a luggage, the first determination unit 104 identifies the luggage space where the person is placing the luggage and then requests the second determination unit 105 to verify the owner against the location of the luggage space. The location (partition) of the luggage space may be determined from the position in the image of camera C, or from the partition information of the luggage space captured in the image.
[0042] When the second determination unit 105a receives a verification request from the first determination unit 104, it refers to the management information held in the management unit 107a and determines whether the combination of the person and the luggage space is a combination that has been registered in advance. If the determination determines that the combination of the person and the luggage space is not a combination that has been registered in advance, the second determination unit 105a requests the notification unit 106a to perform a notification process to inform the notification of a misplacement or unauthorized use by someone who has not made a reservation.
[0043] When the notification unit 106a receives a request for a notification process to notify of misplacement of luggage or unauthorized use by an unreserved person from the second determination unit 105, it performs a predetermined notification operation to notify of the misplacement of luggage or unauthorized use by an unreserved person. For example, the notification unit 106a notifies the person attempting to place the luggage that they are attempting to place the luggage in the wrong luggage space. This notification may be, for example, an audio output from a speaker installed in the vehicle or a message output to a display device installed in the vehicle.
[0044] Next, the operation of the present embodiment will be described in detail with reference to the drawings. FIG. 9 is a diagram for explaining the operation of the present disclosure. The operations up to steps S101 to S103 in FIG. 9 are the same as those in the first embodiment, so the description thereof will be omitted.
[0045] After the estimation of the posture of the person P is completed (step S103), the luggage monitoring device 100a determines whether the person is attempting to place the luggage based on the positional relationship between the luggage B and the person P and the feature amount of the posture of the person P (step S104a).
[0046] If it is determined as a result of the determination that the person is attempting to place the luggage, the luggage monitoring device 100a identifies the luggage space where the luggage is to be placed and then collates it with the reservation details (step S105a). If the luggage space where the person P is attempting to place the luggage matches the luggage space registered in the management information, the luggage monitoring device 100a does nothing.
[0047] On the other hand, when the luggage space where the person P is about to place the luggage does not match the luggage space registered in the management information, the luggage monitoring device 100a performs a predetermined notification operation to notify of the incorrect placement of the luggage (step S106a). For example, as shown in the lower right balloon in FIG. 6, the luggage monitoring device 100 notifies the person or the administrator of the vehicle. The person who receives the notification moves the luggage. Also, the administrator of the vehicle who receives the notification rushes to the scene and requests the person to move the luggage to the correct luggage space. In the process of moving the luggage, an action in which the corresponding person tries to take out the luggage will be detected. In this case, it is determined that the correct owner is taking out the luggage.
[0048] As described above, according to the present disclosure, in addition to the misplacement of luggage in the vehicle, it is possible to detect the incorrect placement of luggage and unauthorized use by unreserved persons, and prompt appropriate responses. In the above-described embodiment, it has been described that one luggage space is allocated to a reserved passenger, but two or more luggage spaces may be allocated to one passenger or a group of passengers.
[0049] [Third Embodiment] Next, a third embodiment in which a function of detecting a key unlocking operation or a destruction operation is added to the luggage monitoring device 100 will be described. FIG. 10 is a functional block diagram showing another configuration of the luggage monitoring device 100b of the present disclosure. The difference from the luggage monitoring device 100 of the first embodiment shown in FIG. 4 is that a function of detecting a key unlocking operation or a destruction operation based on the posture of a person is added to the luggage monitoring device 100b. Since other configurations are substantially the same as those of the first embodiment, the difference in its operation will be mainly described below.
[0050] The first determination unit 104b determines whether a person is about to perform a key unlocking operation or key destruction of the luggage based on the positional relationship between the luggage and the person and the feature amount of the person's posture, in addition to the action of the person trying to take out the luggage. When it is determined that the person is about to perform an unlocking operation or key destruction, the first determination unit 104b requests the notification unit 106b to perform a notification process notifying that the person is about to perform a key unlocking operation or key destruction of the luggage.
[0051] When the notification unit 106b receives a request for notification processing from the first determination unit 104b, it performs a predetermined notification operation to inform that someone is attempting to unlock or destroy the lock on the luggage. For example, the notification unit 106b notifies the vehicle manager that there is someone attempting to unlock or destroy the lock on the luggage. This notification may be, for example, output as audio from the speaker of the vehicle manager's terminal, or output as a message on a display device.
[0052] Next, the operation of this embodiment will be described in detail with reference to the drawings. Figure 11 is a diagram illustrating the operation of this disclosure. The operation from steps S101 to S103 in Figure 11 is the same as in the first embodiment, so its description will be omitted.
[0053] After the estimation of person P's posture is completed (step S103), the luggage monitoring device 100b determines whether the person is attempting to unlock or destroy the luggage lock based on the positional relationship between luggage B and person P and the characteristic quantities of person P's posture (step S104b).
[0054] If the determination results in the finding that person P is attempting to unlock or destroy the lock on the luggage, the luggage monitoring device 100b performs a notification operation to inform the public that there is a person attempting to unlock or destroy the lock on the luggage (step S106b). For example, as shown in the speech bubble in the lower right of Figure 11, the luggage monitoring device 100b notifies the owner of the luggage or the manager of the vehicle. For example, the manager of the vehicle who receives the notification rushes to the scene and takes necessary action, such as speaking to the person in question.
[0055] As explained above, this disclosure makes it possible to detect unintended unlocking or destruction of luggage locks inside a vehicle and prompt appropriate action.
[0056] While the embodiments of this disclosure have been described above, this disclosure is not limited to the embodiments described above, and further modifications, substitutions, and adjustments can be made without departing from the basic technical concept of this disclosure. For example, the network configurations, element configurations, and data representations shown in the drawings are examples to aid in understanding this disclosure and are not limited to the configurations shown in these drawings.
[0057] Furthermore, in the first to third embodiments described above, the luggage monitoring devices 100 to 100b were described as estimating the actions of a person using features extracted from their image. However, methods for estimating actions without using features may also be employed. For example, the luggage monitoring devices 100 to 100b may estimate the actions of a person captured by the camera by determining similarity with images of people attempting to take out luggage that have been registered in advance. Alternatively, a large-scale language model (LLM) or very large language model (VLM) capable of performing image analysis can be used for this action estimation.
[0058] Furthermore, in the first to third embodiments described above, it was explained that the luggage monitoring devices 100 to 100b hold management information within the management units 107 to 107b, but the management information may also be held on a server or the like that is accessible from the luggage monitoring devices 100 to 100b.
[0059] Furthermore, although the drawings illustrating the first to third embodiments described above use suitcases and carry-on bags as examples, the luggage monitored by the luggage monitoring devices 100 to 100b is not limited to these. For example, the luggage may be a stroller, cart, pet cage, etc.
[0060] Furthermore, in the first to third embodiments described above, the information of the luggage image received from the passenger and the passenger (boarder) image are combined into one, but they may be stored as separate data. The luggage image received from the passenger may be saved by performing object recognition on a photograph taken by the user when they brought in their luggage and attaching tags such as "trunk case" or "stroller". Such tags can be used by the first determination unit 104 for behavior estimation. The passenger (boarder) image may also be saved in the form of information suitable for facial recognition, such as feature quantities extracted from the photograph.
[0061] (Hardware Configuration) In each embodiment of this disclosure, each component of each device represents a functional unit block. Some or all of each component of each device is realized by any combination of an information processing device 900 and a program, for example, as shown in Figure 12. Figure 12 is a block diagram showing an example of the hardware configuration of the information processing device 900 that realizes each component of each device. The information processing device 900 includes, as an example, the following configuration: ・CPU (Central Processing Unit) 901 ・ROM (Read Only Memory) 902 ・RAM (Random Access Memory) 903 ・Program 904 loaded into RAM 903 ・Storage device 905 that stores the program 904 ・Drive device 907 that reads and writes to the recording medium 906 ・Communication interface 908 that connects to a communication network 909 ・Input / output interface 910 that performs data input and output ・Bus 911 that connects each component
[0062] Each component of each device in each embodiment is realized by the CPU 901 acquiring and executing a program 904 that realizes these functions. That is, the CPU 901 in Figure 12 executes a person detection program and an action estimation program, and performs update processing of each calculation parameter held in RAM 903, storage device 905, etc. The program 904 that realizes the functions of each component of each device is, for example, stored in storage device 905 or ROM 902 in advance and read by the CPU 901 as needed. The program 904 may be supplied to the CPU 901 via a communication network 909, or it may be stored in a recording medium 906 in advance, and the drive device 907 may read the program and supply it to the CPU 901.
[0063] There are various variations in how each device is implemented. For example, each device may be implemented by any combination of a separate information processing device 900 and a program for each component. Alternatively, multiple components of each device may be implemented by any combination of a single information processing device 900 and a program. That is, each part (processing means, function) of the above-described luggage monitoring device can be implemented by a computer program that causes a processor mounted on the device to execute the above-described processes using its hardware.
[0064] Furthermore, some or all of the components of each device are realized by other general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be made up of a single chip or multiple chips connected via a bus.
[0065] Some or all of the components of each device may be realized by a combination of the circuits and programs described above.
[0066] When some or all of the components of each device are implemented by multiple information processing devices or circuits, these multiple information processing devices or circuits may be centrally located or distributed. For example, the information processing devices or circuits may be implemented in a form in which each is connected via a communication network, such as a client-and-server system or a cloud computing system.
[0067] The embodiments described above are preferred embodiments of this disclosure and do not limit the scope of this disclosure to these embodiments alone. That is, a person skilled in the art can modify or substitute the embodiments described above to construct various modified forms without departing from the gist of this disclosure.
[0068] Some or all of the above embodiments may also be described as follows, but are not limited to these.
[0069] [Note 1] A luggage monitoring device comprising: an acquisition means capable of acquiring images from a camera capable of photographing the luggage storage area inside a vehicle; a detection means for detecting a person touching luggage captured in the camera's image; an estimation means for estimating the posture of the person touching the luggage; a first determination means for determining the person's actions based on the positional relationship between the luggage and the person and the person's posture; a second determination means for determining whether the combination of the person and the luggage is a pre-registered combination if it is determined that the person is attempting to take the luggage; and a notification means for performing a predetermined notification action if it is determined that the combination of the person and the luggage is not a pre-registered combination. [Note 2] The above luggage monitoring device may further be configured to include a management means for managing the passenger's image and the luggage's image in association. [Note 3] The above luggage monitoring device may further be configured to include a reception means for receiving registration of management information, including the passenger's image and the luggage's image, from the passenger at a predetermined timing before boarding. [Note 4] The receiving means of the above-described baggage monitoring device may be configured to accept registration of images of the passenger's companions in addition to the passenger's image. [Note 5] The receiving means of the above-described baggage monitoring device may be configured to assign the passenger a place to place the luggage, and the notification means may be configured to perform a predetermined notification action if the passenger places the luggage anywhere other than the assigned place. [Note 6] The first determination means of the above-described baggage monitoring device may be configured to estimate the action the person is about to take by determining similarity with a previously registered image of the person attempting to take the luggage. [Note 7] The image used by the above-described baggage monitoring device when estimating the person's action may be an image taken of a pre-selected model person performing the action of attempting to take the luggage. [Note 8] The first determination means of the above-described baggage monitoring device may be configured to estimate the action by determining similarity between the characteristic features of the person's posture extracted from the image of the person attempting to take the luggage and the characteristic features of the posture of the human body captured by the camera.[Note 9] The first determination means of the luggage monitoring device described above can determine whether the person is performing an unlocking operation or a lock-breaking operation on the luggage based on the luggage and the posture of the person, and the notification means can be configured to make a notification to a recipient that includes whether the person is performing an unlocking operation or a lock-breaking operation on the luggage. [Note 10] The estimation means of the luggage monitoring device described above can be configured to estimate the action that the person is about to take using a large-scale language model capable of handling images. [Note 11] A luggage monitoring method comprising: acquiring an image from a camera capable of photographing the luggage storage area inside a vehicle; detecting a person touching luggage in the image from the camera; estimating the posture of the person touching the luggage; determining the person's action based on the positional relationship between the luggage and the person and the person's posture; determining whether the combination of the person and the luggage is a pre-registered combination if it is determined that the person is about to take the luggage; and performing a predetermined notification operation if it is determined that the combination of the person and the luggage is not a pre-registered combination. [Note 12] A recording medium that records a program causing a computer to execute the following: a process of acquiring an image from a camera capable of photographing the luggage storage area inside the vehicle; a process of detecting a person touching luggage captured in the camera's image; a process of estimating the posture of the person touching the luggage; a process of determining the person's actions based on the positional relationship between the luggage and the person, and the person's posture; a process of determining whether the combination of the person and the luggage is a pre-registered combination if it is determined that the person is attempting to take the luggage; and a process of performing a predetermined notification action if it is determined that the combination of the person and the luggage is not a pre-registered combination. The forms described in each of the above notes can be combined with each other after making the necessary modifications. For example, a configuration that combines the contents described in Note 2 and the contents described in Note 3 is also included in the scope of disclosure of this specification. In this case, the luggage monitoring device will be configured to accept registration of management information including the passenger's image and the luggage's image from the passenger at a predetermined timing before boarding, and to manage it as management information.Furthermore, the forms described in appendices 10 to 11 above can be expanded into the forms described in appendices 2 to 9, similar to appendice 1.
[0070] Furthermore, each disclosure in the above-mentioned patent documents is incorporated into this document by reference and may be used as the basis or part of this disclosure as necessary. Within the framework of this disclosure (including the claims), further modifications and adjustments to the embodiments or examples are possible based on their fundamental technical concept. Also, within the framework of this disclosure, various combinations or selections (including partial deletions) of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible. In other words, this disclosure naturally includes the entire disclosure, including the claims, and various modifications and alterations that a person skilled in the art could make in accordance with the technical concept. In particular, with respect to the numerical ranges described in this document, any numerical value or sub-range included within that range should be interpreted as being specifically described, even if not otherwise stated. Furthermore, each disclosure item of the above-mentioned cited documents may, as necessary, be used in part or in whole as part of this disclosure, in accordance with the spirit of this disclosure, and this is also considered to be included in the disclosure items of this application.
[0071] 10, 100, 100a, 100b Luggage monitoring device 11 Acquisition means 12 Detection means 13 Estimation means 14 First determination means 15 Second determination means 16 Notification means 101 Acquisition unit 102 Detection unit 103 Estimation unit 104, 104a, 104b First determination unit 105, 105a Second determination unit 106, 106a, 106b Notification unit 107, 107a Management unit 108, 108a Reception unit 200 Mobile terminal 900 Information processing device 901 CPU (Central Processing Unit) 902 ROM (Read Only Memory) 903 RAM (Random Access Memory) 904 Program 905 Storage device 906 Recording medium 907 Drive device 908 Communication interface 909 Communication network 910 Input / output interface 911 Bus C Camera N Network
Claims
1. A luggage monitoring device comprising: an acquisition means capable of acquiring an image from a camera capable of photographing the luggage storage area inside a vehicle; a detection means for detecting a person touching luggage captured in the image from the camera; an estimation means for estimating the posture of the person touching the luggage; a first determination means for determining the actions of the person based on the positional relationship between the luggage and the person and the posture of the person; a second determination means for determining whether the combination of the person and the luggage is a pre-registered combination if it is determined that the person is attempting to take the luggage; and a notification means for performing a predetermined notification action if it is determined that the combination of the person and the luggage is not a pre-registered combination.
2. The luggage monitoring device according to claim 1, further comprising a management means for linking and managing the passenger's image with the luggage's image.
3. The luggage monitoring device according to claim 1 or 2, further comprising a receiving means for receiving registration of management information, including an image of the passenger and an image of the luggage, from the passenger at a predetermined timing before boarding.
4. The luggage monitoring device according to claim 3, wherein the receiving means accepts registration of images of the passenger's companions in addition to the passenger's image.
5. The luggage monitoring device according to claim 3 or 4, wherein the receiving means assigns a place to place the luggage to the passenger, and the notification means performs a predetermined notification action if the passenger places the luggage in a location other than the assigned place.
6. The luggage monitoring device according to any one of claims 1 to 5, wherein the first determination means estimates the action that a person is about to take by determining similarity with an image of a person who is about to take out luggage that has been registered in advance.
7. The luggage monitoring device according to claim 6, wherein the image of the person attempting to take out the luggage is an image taken by having a pre-selected model person perform the action of attempting to take out the luggage.
8. The luggage monitoring device according to claim 6 or 7, wherein the first determination means estimates the action by determining the similarity between the characteristic features of the posture of the person extracted from an image of the person taking out the luggage and the characteristic features of the posture of the human body captured by the camera.
9. The luggage monitoring device according to any one of claims 1 to 8, wherein the first determination means is capable of determining whether the person is performing an unlocking operation or a lock-breaking operation on the luggage, based on the luggage and the posture of the person, and the notification means makes a notification to a recipient including whether the person is performing an unlocking operation or a lock-breaking operation on the luggage.
10. A control system according to any one of claims 1 to 9, wherein the estimation means estimates the action that the person is about to take using a large-scale language model capable of handling images.
11. A luggage monitoring method comprising: acquiring images from a camera capable of photographing the luggage storage area inside a vehicle; detecting a person touching luggage in the camera's images; estimating the posture of the person touching the luggage; determining the person's actions based on the positional relationship between the luggage and the person and the person's posture; determining whether the combination of the person and the luggage is a pre-registered combination if it is determined that the person is attempting to take the luggage; and performing a predetermined notification action if it is determined that the combination of the person and the luggage is not a pre-registered combination.
12. A recording medium that contains a program that causes a computer to execute the following: a process of acquiring an image from a camera capable of photographing the luggage storage area inside a vehicle; a process of detecting a person touching luggage in the image from the camera; a process of estimating the posture of the person touching the luggage; a process of determining the actions of the person based on the positional relationship between the luggage and the person and the posture of the person; a process of determining whether the combination of the person and the luggage is a pre-registered combination if it is determined that the person is trying to take the luggage; and a process of performing a predetermined notification action if it is determined that the combination of the person and the luggage is not a pre-registered combination.