Station yard maintenance system, station yard maintenance method, and station yard maintenance program

Abstract

To provide a system which can take a necessary countermeasure against a person who exists in a station yard and against whom some countermeasure is required.SOLUTION: A station yard maintenance system 1 comprises an image recognition unit 22a, a command generation unit 22c, and a robot 30. The image recognition unit 22a successively acquires images generated by capturing a station yard with an imaging apparatus 11, with the lapse of time and specifies information of a person included in the images by image recognition. The command generation unit 22c determines a countermeasure against the person on the basis of information of the person specified by the image recognition unit 22a and generates a command to take the countermeasure. The robot 30 receives the command generated by the command generation unit 22c and moves to a position of the person to take the countermeasure against the person on the basis of the command.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present application primarily relates to a system for dealing with people in a station premises and maintaining the environment within the station premises. [Background technology]

[0002] Patent Document 1 discloses a manners violation detection system. This system includes a sensor that detects manners violations, a camera that captures images of the station premises, and a station work remote monitoring system. The station work remote monitoring system is a system in which station staff monitor the station from a remote location. When the sensor detects a manners violation, the sensor transmits an alarm signal to the monitoring system. This allows station staff of the station work remote monitoring system to know that a manners violation has occurred. In addition, the monitor system display shows an image obtained from a camera close to the sensor that detected the manners violation. This allows station staff to know the details of the manners violation. Patent Document 1 describes smoking and littering as examples of manners violations. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2002-232871 A Summary of the Invention [Problem to be solved by the invention]

[0004] The etiquette violation detection system of Patent Document 1 can detect etiquette violations, but cannot deal with them. Station staff must deal with etiquette violations. However, station staff have many tasks to perform, and may not be able to spare the time to deal with etiquette violations. Furthermore, this issue is not limited to dealing with etiquette violations, but is a common issue for all work that station staff do within the station.

[0005] This application has been made in consideration of the above circumstances, and its main objective is to provide an in-station maintenance system that enables the system to take the necessary action for people within a station who require some kind of attention. [Means for solving the problem]

[0006] The problem to be solved by the present application is as described above. Next, the means for solving this problem and the effects thereof will be described.

[0007] According to a first aspect of the present application, there is provided an in-station maintenance system having the following configuration. That is, the in-station maintenance system includes an image recognition unit, a command generation unit, and a robot. The image recognition unit sequentially acquires images generated by capturing images of the inside of the station with an imaging device over time, and identifies information of people appearing in the images by image recognition. The command generation unit determines how to deal with the person based on the information of the person identified by the image recognition unit, and generates a command for the action. The robot receives the command generated by the command generation unit, moves to the position of the person based on the command, and deals with the person.

[0008] According to a second aspect of the present application, there is provided the following in-station maintenance method. That is, in the in-station maintenance method, images generated by capturing images of the inside of a station with an imaging device are sequentially acquired over time, and information on a person appearing in the images is identified by image recognition. In the in-station maintenance method, a measure according to the person is determined based on the information on the person identified by the image recognition unit, and a command for the measure is generated and transmitted to a robot. In the in-station maintenance method, the robot that has received the command is moved to the position of the person based on the command, and the robot is made to deal with the person.

[0009] According to a third aspect of the present application, there is provided a station premises maintenance program that causes a computer to execute the following process. That is, images generated by capturing images of the station premises with an imaging device are sequentially acquired over time, and information on people appearing in the images is identified by image recognition. Based on the information on the people identified by the image recognition unit, a countermeasure according to the people is determined, and a command for the countermeasure is generated and transmitted to a robot. Effect of the Invention

[0010] According to the present application, the system can take the necessary action for a person who is in a station and requires some kind of action. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram of a station maintenance system and related devices according to an embodiment of the present application; [Diagram 2] 13 is a flowchart of a process performed by a processing server. [Diagram 3] 1 is a diagram showing an overview of information used to estimate a person's behavior or situation and how to handle it, and the estimation results. [Figure 4] 13 is a diagram showing an overview of information used to estimate the behavior or situation of a suspicious person and how to deal with the situation, and the estimation results. [Diagram 5] A diagram showing an overview of the information used to estimate the behavior or situation of a general passenger and how to respond, and the estimation results. [Figure 6] FIG. 13 is a diagram showing an overview of the information used to estimate the behavior or situation of a person requiring care and how to deal with the situation, and the estimation results. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Next, an embodiment of the present application will be described with reference to the drawings.

[0013] The station premises maintenance system 1 is a system that reduces the burden on station staff by performing some of the work that station staff currently perform on their behalf. Specifically, the station premises maintenance system 1 acquires information such as images of the station premises, identifies necessary measures based on the acquired information, and performs the necessary measures using a robot.

[0014] FIG. 1 shows an imaging device 11 and a railway information DB 13 as components for providing information within a station.

[0015] The imaging device 11 captures images of the station premises, generates images, and stores them. The images generated by the imaging device 11 are visible images obtained by converting visible light into images, and are not thermal images or distance images. The imaging device 11 may be fixed within the station premises, or may be movable or capable of changing the imaging angle. When the imaging device 11 is movable or capable of changing the imaging angle, it is preferable to store position information or orientation information together with the images. The imaging device 11 may be a camera whose main function is to capture still images, or a video camera whose main function is to capture videos. The imaging device 11 may also be a camera module incorporated in another device. Specifically, the imaging device 11 is a station premises camera, a robot camera, or a smartphone camera. The station premises camera is a camera fixed to a ceiling or a pillar, etc., within the station premises, and may be used as a security camera, etc. The robot camera is a camera module incorporated in a robot 30 described later. The smartphone camera is a camera module incorporated in a smartphone.

[0016] The imaging device 11 further has a communication function. The imaging device 11 transmits an image by wired communication or wireless communication. The image transmitted by the imaging device 11 is uploaded to the cloud via a PC or a router in the station. Alternatively, if the imaging device 11 is incorporated in a smartphone, the smartphone may upload the image to the cloud via a mobile network. It is preferable that the imaging device 11 transmits an image in association with information specifying the position or orientation of the image. For example, it is preferable to transmit an image in association with the above-mentioned position information or orientation information. Alternatively, if the identification information of the imaging device 11 and the position of the imaging device 11 are registered separately in association with each other, it is preferable to transmit an image in association with the identification information of the imaging device 11.

[0017] In addition to the imaging device 11, a sensor that detects information other than images may be provided. For example, a heat sensor, a sound sensor, a three-dimensional sensor, or a vibration sensor may be provided. Sensor information detected by these sensors may be used to estimate the situation within the station, for example, the behavior or situation of a person.

[0018] The railway information DB13 is a database that stores information on stations, facilities, passengers, operation, route maps, etc. Specific information will be described later. The railway information DB13 is stored in a PC within the station or a server outside the station, etc. The railway information DB13 includes information received from outside the station via the Internet, etc., information acquired at the station via a sensor or ticket gate 14, etc., information manually entered by station staff, etc.

[0019] The station premises maintenance system 1 comprises an image storage server 21 and a processing server 22 on the cloud. On the cloud refers to an environment constructed with servers and the like that are connected via the Internet and are located physically away from the station premises. Note that the image storage server 21 and the processing server 22 are not limited to being on the cloud, and may be provided in the station premises. Also, the image storage server 21 may be realized by one server device, or may be realized by a plurality of server devices working together. The same applies to the processing server 22.

[0020] The image storage server 21 stores images and the like transmitted by the imaging device 11. The image storage server 21 stores images and transmits them to the processing server 22. The image storage server 21 is not an essential component and can be omitted.

[0021] The processing server 22 generates a command to identify a necessary measure and transmit it to the robot 30 based on the image of the station premises acquired from the image storage server 21, the information acquired from the railway information DB, and the public information. Here, the public information is information available on the Internet and is used to analyze the information of the station premises. The processing server 22 is a server device including a CPU, a storage device, a memory, a communication device, and the like. The processing server 22 realizes various functions by the CPU executing a program stored in the storage device. Specifically, the processing server 22 has functions as an image recognition unit 22a, an analysis unit 22b, and a command generation unit 22c. In this embodiment, one processing server 22 has the functions of the image recognition unit 22a, the analysis unit 22b, and the command generation unit 22c, but this is an example, and a different server device may be used for each function.

[0022] The image recognition unit 22a performs image recognition on the image captured inside the station to identify the behavior or situation and position change of the person shown in the image. The behavior or situation of the person includes not only the current behavior or situation but also the predicted future behavior or situation. In this embodiment, the necessary measures for the person are identified based on the behavior or situation of the person. The position change of the person is information indicating the change in the position of the identified person. The details of the processing of the image recognition unit 22a will be described later.

[0023] The analysis unit 22b analyzes the sensor information when a sensor is installed in the station. The analysis result of the sensor information is used to specify the behavior or situation of a person in more detail, or to specify the situation in the station in more detail. For example, the analysis unit 22b can specify that a person is likely to be smoking by identifying a cigarette based on the detection result of a heat sensor. The analysis unit 22b can specify that a person is likely to be unfamiliar with riding a train by recognizing a foreign language based on the detection result of a sound sensor. The analysis unit 22b can specify the congestion situation in the station more accurately based on the detection result of a three-dimensional sensor or a vibration sensor. The information analyzed by the analysis unit 22b may be used to assist the image recognition of the image recognition unit 22a, or may be used to assist the command generation of the command generation unit 22c.

[0024] The command generating unit 22c generates a command for dealing with the person based on the behavior or situation of the person identified by the image recognizing unit 22a and the position change, and transmits the command to the robot 30. The process of the command generating unit 22c will be described in detail later.

[0025] The robot 30 is a general-purpose robot capable of autonomous travel and various tasks. A plurality of robots 30 are installed within the station. Among the plurality of robots 30, there are robots 30 with different functions. The robot 30 includes a detection sensor 31, a control device 32, a work tool 33, a travel drive unit 34, a display unit 35, and a speaker 36.

[0026] The detection sensor 31 detects the surroundings of the robot 30. The detection sensor 31 is a camera, a microphone, a radar, a LiDAR, or an ultrasonic sensor. By traveling based on the detection result of the detection sensor 31, the robot 30 does not collide with passengers or pillars. The detection sensor 31 is also used to identify the positional relationship with a person to be dealt with and to obtain information about the person. The control device 32 is a microcomputer having a CPU, storage, memory, communication device, etc., and controls the work tool 33, the travel drive unit 34, the display unit 35, and the speaker 36 based on the received command. This allows the robot 30 to take the necessary measures.

[0027] The work tool 33 is a tool used by the robot 30 to perform work. The work tool 33 may be a dedicated tool for a specific task. An example of a dedicated tool is a tool for caring for a person in need of care. The work tool 33 may also be a general-purpose tool that can be used for various tasks. An example of a general-purpose tool is a robot hand. The robot 30 may also be equipped with a tool changer. This allows the robot 30 to handle various tasks by switching the work tool 33.

[0028] The traveling drive unit 34 is a motor that drives the wheels, crawlers, legs, etc. to make the robot travel. The control device 32 controls the traveling drive unit 34 according to the target position that is also included in the command. This allows the robot 30 to move to the target position.

[0029] The display unit 35 is a liquid crystal display or an organic EL display, and is capable of displaying information. The control device 32 generates an image based on the information included in the command and displays it on the display unit 35. This makes it possible to notify information to passengers, station staff, etc.

[0030] The speaker 36 converts the electrical signal into sound and emits the sound. The control device 32 generates sound based on the information included in the command and outputs the sound from the speaker 36. This makes it possible to notify information to passengers, station staff, etc.

[0031] Next, the flow of processing performed by the station premises maintenance system 1 will be described with reference to the flowchart shown in FIG.

[0032] First, when the processing server 22 becomes ready for processing, it accesses the image storage server 21 and acquires images of the station premises (S101). In the station premises maintenance system 1, in order to handle people over a wide range in the station premises, it is preferable for the processing server 22 to acquire images of various positions. Also, in order to acquire positional changes or movements of people, it is preferable for the processing server 22 to acquire images of the same shooting angle at a predetermined time interval.

[0033] The processing server 22, more specifically the image recognition unit 22a, performs image recognition on the image of the station premises and extracts images of people (S102). The image recognition unit 22a recognizes and extracts people included in the image of the station premises, for example, by matching processing or AI processing. When performing matching processing, the image recognition unit 22a recognizes people included in the image of the station premises by searching for a pattern image showing a human shape from the image of the station premises. When performing AI processing, a judgment model is created in advance by machine learning various human images. Then, at the time of estimation, the image recognition unit 22a inputs the image of the station premises to be judged into the judgment model, and the range of people included in the image of the station premises, etc. are output.

[0034] Next, the image recognition unit 22a performs image recognition on one or more images of the person to identify the following information (S103). The multiple images are images taken at the same shooting angle but different shooting times. However, to improve the estimation accuracy, images taken at different shooting angles but at the same shooting time may be used. The information specifically identified by the image recognition unit 22a is, for example, the facial expression of the person, the posture of the person, the movement and position change of the person, the possessions of the person, the age, sex, physique, etc. of the person. The image recognition unit 22a identifies these pieces of information by, for example, matching processing or AI processing. The basic processing is the same as step S102, but the data handled is different. That is, in the case of matching processing, a pattern image according to the information to be identified is used. In the case of AI processing, a judgment model is created using learning data according to the information to be recognized.

[0035] Thereby, the image recognition unit 22a can identify the facial expression of the person, such as troubled, angry, crying, etc. The image recognition unit 22a can identify the posture of the person, such as bending over, raising hands, etc. The image recognition unit 22a can identify the possessions of the person, such as holding a walking stick, holding a large baggage, holding a knife, etc.

[0036] The image recognition unit 22a identifies the movement and position change of a person based on a plurality of images of the person taken at different times. In other words, the movement of a person is a change in the person's posture. Therefore, the image recognition unit 22a identifies the movement of a person by identifying the posture of the person through image recognition for a plurality of images of the person taken at different times. As a premise, the image recognition unit 22a needs to be able to identify the same person from a plurality of images taken at different times. Here, the image recognition unit 22a identifies various features of a person through image recognition of the images of the person, and therefore can identify the same person with a high degree of accuracy based on the degree of matching of the features.

[0037] This allows the image recognition unit 22a to identify the person's movements as walking, running, frequently looking left and right, moving their arms widely, and the like.

[0038] The image recognition unit 22a identifies the position change of the person as follows. In this embodiment, the position of the person is identified using an area number. The area number is identification information for identifying each area when the station premises is divided into a plurality of areas. The image recognition unit 22a identifies the position of the person in the station premises based on the position of the imaging device 11 that captured the image and the position of the person on the image. As described above, the position of the imaging device 11 that captured the image is stored in association with the image. The position of the person on the image can be identified based on the positional relationship between the image and the person. Since the approximate position of the person can be identified based on these pieces of information, the area number indicating the position of the person can be identified. Furthermore, the image recognition unit 22a identifies the position change of the person by identifying the position of the person by image recognition for a plurality of images of the person captured at different times. Here, the image recognition unit 22a also uses a process for identifying the same person from a plurality of images captured at different times.

[0039] This allows the image recognition unit 22a to identify a change in area number according to time as a change in the person's position. Since the image of the station premises is a two-dimensional image, it is difficult to accurately identify the three-dimensional position of a person. However, in this embodiment, the robot 30 ultimately identifies the position of the person, so it is sufficient to be able to identify the approximate position of the person. The information described in step S103 is an example. Therefore, information not described in this specification may be identified by image recognition, and identification of some of the information described in this specification may be omitted.

[0040] Next, the image recognition unit 22a estimates the behavior or situation of the person based on the information identified by the image recognition (S104). The image recognition unit 22a compares the identified information with a database created in advance to estimate the behavior or situation of the person. For example, the database associates information on a person that can be identified by image recognition with the behavior or situation of the person. Three examples are shown below. In the first example, the information identified by image recognition is "the person's position is near the vehicle passage position on the platform" and "the person frequently looks left and right." The behavior or situation associated with this is "a sign of a collision between a person and a train." In the second example, the information identified by image recognition is "a person has a troubled expression" and "a person stands at a ticket machine or information board for a long time." The behavior or situation associated with this is "a state of requesting information regarding boarding a train." In the third example, the information identified by image recognition is "the person's possession is a cane" and "the person's posture is bent over." The behavior or situation associated with this is "a situation in which assistance is required for the person's movement." The configuration of the database is an example, and three or more pieces of information identified by image recognition may be associated with a person's behavior or situation. In this case, for example, when two or more pieces of information match, it may be determined that the behavior or situation is associated. The database may also include sensor information.

[0041] The image recognition unit 22a can also estimate the behavior or situation of a person by AI processing. In this case, learning data is created that associates information identified by image recognition with the behavior or situation of a person, and this learning data is machine-learned to create a judgment model in advance. The above-mentioned database is created by extracting some information that is clearly related to a specific behavior or situation of a person from the information identified by image recognition and storing it in a database. In contrast, machine learning can learn whether or not data is related to each other and the level of the relationship, so all information identified by image recognition may be included in the learning data. At the time of estimation, the information identified by image recognition is input into the judgment model, and an estimated result of the behavior or situation of a person is output.

[0042] Next, the processing server 22, more specifically the command generation unit 22c, estimates a necessary measure to be taken by the robot 30 based on the person's behavior or situation and the railway information (S105). The command generation unit 22c compares the person's behavior or situation estimated in step S104 with a database created in advance to estimate the necessary measure. For example, the database associates the person's behavior or situation with the necessary measures. Three examples are shown below. In the first example, the behavior or situation of a person is "a sign of jumping in front of a train." The action associated with this is "the robot 30 moves to the position of the person and speaks to the person." In the second example, the person's behavior or situation is "a state in which information is being sought regarding riding in a railroad car," and the corresponding action is "the robot 30 moves to the person's location and provides the person with information regarding riding in a railroad car." In the third example, the behavior or situation of the person is "a situation in which assistance is required for the person's movement." The corresponding action is "the robot 30 moves to the person's position and assists the person's movement." The configuration of the database is an example, and multiple measures may be associated with each of them, and any one of them may be selected depending on the situation. The database may also include sensor information.

[0043] The command generating unit 22c can also estimate the necessary measures by AI processing. In this case, only the above-mentioned database may be machine-learned, but it is preferable to create a judgment model by machine-learning other related information, such as sensor information or the position of a person. Then, at the time of estimation, the command generating unit 22c inputs the behavior or situation of a person and the detected or estimated related information into the judgment model, and an estimated result of the necessary measures is output.

[0044] In this manner, in this embodiment, an image of a person is extracted (S102), information about the person is identified (S103), the person's behavior or situation is estimated (S104), and a response according to the behavior or situation is estimated (S105). It is not necessary to perform all of these processes, and some of the processes can be omitted. For example, when the identification of the person's information (S103) is omitted, a judgment model is created by machine learning data that associates the image of the person with the behavior or situation of the person. Then, at the time of estimation, the image recognition unit 22a inputs the image of the person to be judged into the judgment model, and the behavior or situation of the person is output. The same applies when other processes are omitted. Also, multiple processes may be omitted.

[0045] When estimating the person's behavior or situation (S104) and estimating the measures according to the person's behavior or situation (S105), the above-mentioned railway information may be further used. This allows the person's behavior or situation to be accurately estimated and the necessary measures to be appropriately estimated. The specific contents of the railway information and how to use it will be described later.

[0046] Next, the command generating unit 22c generates a command to take necessary measures based on the change in the person's position and the necessary measures (S107). The change in the person's position is used to command the robot 30 to move to the position of the target person. For example, if the person moves and the area number changes, the command generating unit 22c notifies the robot 30 of the area number after the move. In addition, the command generating unit 22c may transmit the characteristics of the person to the robot 30 so that the robot 30 can identify the target person.

[0047] Next, the command generating unit 22c selects a robot 30 capable of executing the generated command (S107). As described above, a plurality of robots 30 with different functions are provided in the station. The area in which the robot 30 can move is determined in advance. The command generating unit 22c selects a robot 30 capable of executing the generated command by accessing a database in which the identification information of the robot 30, the function of the robot 30, and the area in which the robot 30 can move are associated with each other. The database may also include the current position of the robot 30. In this case, the command generating unit 22c may select a robot 30 such that the position where the action is to be taken is close to the current position of the robot 30.

[0048] Next, the command generating unit 22c transmits a command to the selected robot 30 to have it deal with the target person (S108). The control device 32 of the robot 30 drives the traveling drive unit 34 based on the position of the person included in the received command, specifically the above-mentioned area number, to move to the position indicated by the area number. The robot 30 recognizes the target person based on the characteristics of the person received from the command generating unit 22c and the sensor information detected by the detection sensor 31, and moves to the position of the target person. Thereafter, the control device 32 of the robot 30 controls the work tool 33, the display unit 35, the speaker 36, etc. based on the command received from the command generating unit 22c to take the necessary measures.

[0049] The outline of the response is determined by the command generating unit 22c, but the specific response may be determined by the command generating unit 22c or by the robot 30. For example, the outline of calling out to the target person is determined by the command generating unit 22c, but the content of the call may be determined by the command generating unit 22c or by the robot 30. In addition, when the robot 30 receives a question from a person, the robot 30 may generate a response and output it as a voice, or may transmit a question to the command generating unit 22c and output the response received from the command generating unit 22c as a voice. In other words, various processes related to the response may be handled by either the command generating unit 22c or the robot 30.

[0050] Next, the processing server 22 notifies the station staff of the information that satisfies the notification condition (S109). The notification condition is determined in advance based on the importance or urgency, etc. The information to be notified may be the estimated behavior or situation of the person, or may be the action taken by the robot 30. The notification to the station staff is, for example, transmitted to an information device owned or managed by the station staff. The notification to the station staff may be performed by the robot 30 instead of by the processing server 22.

[0051] Next, the process of estimating the necessary measures from the information identified by image recognition will be described in more detail. First, the information used for the estimation and the estimation result will be outlined with reference to Fig. 3, and then specific examples will be described with reference to Figs. 4 to 6.

[0052] As shown in Fig. 3, the information used for estimation is divided into railway information and information identified by image recognition. Information identified by image recognition is hereinafter referred to as image recognition information. Note that sensor information may be additionally used.

[0053] Railway information is a general term for various information related to railways. Railway information includes station premises information, passenger information, route information, and operation information. Station premises information is information about facilities provided in the station premises, such as information about passageways, platforms, waiting rooms, shops, and equipment. The equipment includes ticket vending machines, ticket gates 14, escalators, elevators, coin lockers, and the like. The location, usage status, and suspension status of each facility are also included in the station premises information. Passenger information is information about passengers, and is information about passengers registered by station staff or the system for some reason. In addition, passenger information includes not only individual passengers but also congestion status. Route information is information about a route map of railway vehicles. Operation information is information about the operation of railways available at the station. For example, operation information is information indicating normal, delay, suspension, etc. for each line.

[0054] The station premises maintenance system 1 estimates the behavior or situation of a person that requires some kind of action. Therefore, the behavior or situation of a person estimated by the image recognition unit 22a can be classified into the following four categories, as shown in Fig. 3. That is, it can be classified into four categories: (1) the person's current problematic behavior, (2) the person's future problematic behavior, (3) the person's request for information, and (4) the person's request for physical support.

[0055] The common measures taken for the four categories are, for example, the robot 30 moving to the position of the person, the robot 30 calling out to, guiding, or monitoring the person, the robot 30 notifying station staff or handing the person over to the station staff, and the robot 30 controlling the transportation equipment. The transportation equipment is equipment installed in the station premises and is used by people to move or regulates the movement of people. Specifically, the transportation equipment is platform doors, doors, escalators, elevators, ticket gates 14, and the like. The control of the transportation equipment means transmitting a signal to the transportation equipment to switch between a state in which the transportation equipment moves people and a state in which the transportation equipment does not move people. The movement of the person can be adjusted by controlling the transportation equipment.

[0056] The outline of the measures taken according to the behavior or situation is as follows. (1) For the person's current problematic behavior, the robot 30 takes measures against the person's current problematic behavior. Specifically, the robot 30 calls out to the person to stop the problematic behavior, or takes measures to reduce the impact of the problematic behavior. (2) For the person's future problematic behavior, the robot 30 prevents the person from taking future problematic behavior. Specifically, the robot 30 calls out to the person to stop the problematic behavior, or monitors the person to make it difficult for the person to perform the problematic behavior. (3) If the person requests information, the robot 30 provides the information requested by the person. Specifically, if the information requested by the person can be estimated, the robot 30 provides the estimated information. If the information requested by the person cannot be estimated, the robot 30 calls out to the person to find out the information requested, and provides the information requested. (4) If the person requests physical support, the robot 30 provides the physical support requested by the person. Specifically, the robot 30 having a function corresponding to the physical support requested by the person should be selected, and the robot 30 provides the physical support using the work tool 33 in response to the command.

[0057] Next, a specific example will be described with reference to Figures 4 to 6. First, with reference to Figure 4, an example of dealing with a person's problematic behavior will be described.

[0058] For example, a suspicious person list in passenger information is a list of information that is particularly used when dealing with problematic behavior. The suspicious person list is a list of people who have engaged in problematic behavior in the past, or people who are manually registered by station staff who have determined that they may engage in problematic behavior. The suspicious person list includes information for identifying a person, such as information about the person's face or skeleton. Therefore, the image recognition unit 22a can estimate whether or not the identified person is registered in the suspicious person list based on the result of image recognition. When the identified person is registered in the suspicious person list, the image recognition unit 22a estimates that the identified person is more likely to engage in problematic behavior than a person not registered in the suspicious person list. Furthermore, when the identified person is registered in the suspicious person list, the command generation unit 22c generates a command because it is necessary to deal with the identified person earlier than a person not registered in the suspicious person list. The process using the suspicious person list is not essential and can be omitted.

[0059] The behavior or situation of a person that is presumed when dealing with problematic behavior includes (a) behavior that may result in a collision with a railroad vehicle, (b) criminal behavior such as violent behavior or molestation, and (c) signs of criminal behavior. Behavior that may result in a collision with a railroad vehicle includes not only behavior that may result in intentionally jumping into a railroad vehicle, but also behavior that may result in a person unintentionally falling onto the tracks and colliding with a railroad vehicle. The possibility of jumping into a railroad vehicle can be presumed using the first example above. The possibility of unintentionally falling onto the tracks can be associated with behaviors such as walking on the edge of a platform or walking unsteadily. Criminal behavior can be easily presumed based on a person's movements. Signs of criminal behavior can be presumed by comprehensively evaluating information such as a person's facial expression, posture, movements, and possessions.

[0060] (a) To deal with a behavior that may result in a collision with a railroad vehicle, for example, the robot 30 moves to the person's position and calls out to or monitors the person. In particular, monitoring is effective when there is a possibility that the person may intentionally jump into a railroad vehicle, and calling out to the person is effective when there is a possibility that the person may unintentionally fall onto the tracks. Furthermore, if the behavior or situation of the person does not change even after the robot 30 calls out to or monitors the person, the command generation unit 22c or the robot 30 notifies a station staff member.

[0061] (b) In response to a criminal act such as a violent act or a molestation act, for example, the robot 30 moves to the location of the person and calls out to the person to stop the criminal act, or monitors the person to stop further criminal acts. If the person escapes, the robot 30 may control the moving equipment to stop the person. Alternatively, the moving equipment may be controlled to prevent the person from moving to a crowded area. The robot 30 may also draw the target person's attention by emitting light from a floodlight as the work tool 33 or by outputting sound from the speaker 36. The command generation unit 22c or the robot 30 notifies a station staff member. An image that serves as evidence of the criminal act may be generated using a camera provided in the robot 30.

[0062] (c) To deal with a sign of a criminal act, for example, the robot 30 moves to the person's location and speaks to or monitors the person. This may prevent the person from committing a criminal act.

[0063] Next, an example of handling general passengers will be described with reference to FIG.

[0064] Examples of information that is particularly used when dealing with general passengers include the congestion status or passenger movement status. The congestion status is information in which a congestion level indicating the degree of congestion is described in association with an area number indicating a position within a station premises. The congestion status can be identified based on the result of image recognition of an image captured and generated by the imaging device 11. Alternatively, it can be identified based on the detection result of the above-mentioned sound sensor or three-dimensional sensor. The flow of passenger movement is information indicating whether passengers are able to move smoothly between two points described using area numbers. The flow of passenger movement is, for example, the number of passengers moving between two points described using area numbers and the movement speed of passengers moving between the two points. The number and movement speed of passengers can be identified based on the result of image recognition of an image captured and generated by the imaging device 11.

[0065] Presumed behaviors or situations of people when dealing with general passengers include: (a) a disruption in service and a request for information on alternative routes, (b) the occurrence or signs of congestion, and (c) a request for information on how to board a train.

[0066] Regarding (a) above, when a delay or suspension occurs on a specific line, using a route that includes that line may result in significant delays or the route may become unusable. For this reason, it is preferable to use a detour route that does not include the line on which the delay or suspension occurred, but passengers may be confused if they do not know the information about the detour route. This situation is estimated based on, for example, conditions that a delay or suspension has occurred based on operation information, that a person is in a location related to the delay or suspension, and that the person's facial expression is distressed. A location related to a delay or suspension is, for example, a platform where a delay or suspension has occurred, or a location where information regarding operation information is available.

[0067] Regarding (b) above, the image recognition unit 22a can estimate that congestion has occurred based on the above-mentioned congestion situation. For example, when the congestion level of the congestion situation is gradually increasing, the image recognition unit 22a estimates that there is a sign of congestion. Alternatively, the image recognition unit 22a can determine the sign of congestion based on information such as a large number of passengers on a train scheduled to arrive at the target station, a large-scale event taking place nearby, etc.

[0068] Regarding (c) above, the image recognition unit 22a can make an estimation based on the second example described above.

[0069] (a) In response to a request for information on a detour route due to a disruption in service, for example, the robot 30 moves to the person's location and presents the detour route using the display unit 35 or the speaker 36. The robot 30 may also guide the passenger to a platform on the detour route.

[0070] (b) A response to the occurrence of congestion or a sign of congestion is, for example, the robot 30 moving to a position related to the occurrence or sign of congestion and guiding or organizing passengers to reduce or prevent congestion. The robot 30 moves to a position where congestion is occurring and guides passengers so that congestion is resolved early. Alternatively, the robot 30 may move to the outside of the crowded area and present a less crowded position or a route that bypasses the crowded area so that passengers do not enter the crowded area. For example, if the crowded area is a specific passage, the robot 30 presents another passage with a lower congestion level. Also, if the crowded area is inside a specific car of a railway vehicle, the robot 30 presents another car with a lower congestion level. The robot 30 may control a moving facility such as a door to make a spare passage available in order to reduce congestion early. When the robot 30 guides or organizes passengers, the above-mentioned information on the passenger movement status is referenced. For example, the robot 30 guides passengers so as not to go against the passengers' movement, or guides passengers to an area where passengers are less likely to move. As another way to deal with the occurrence of congestion, the display unit 35 may display images or the like to relieve passengers of boredom. The command generating unit 22c or the work tool 33 may notify station staff of the mixed loading situation.

[0071] (c) In response to a request for information on how to board a train, for example, the robot 30 moves to the person's location and answers the person's question about how to board the train or presents the person with information on how to board the train. Questions that the robot 30 can answer include, for example, the destination of the train, how to buy a ticket, and how to transfer to the destination.

[0072] Next, with reference to FIG. 6, an example of handling a person requiring assistance will be described.

[0073] For example, coin locker information is station premises information that is particularly used when dealing with persons requiring assistance. Coin locker information is information indicating the location and availability of coin lockers. The location of coin lockers is registered as station premises information. Coin lockers also have a communication function, and the coin lockers transmit their availability status to the railway information database at any time. As a result of the above, coin locker information is created and stored in railway information DB13. Note that information on other facilities, such as elevator information, may be used instead of or in addition to coin locker information.

[0074] Examples of passenger information that is particularly used when dealing with a person requiring assistance include lost child data, IC passenger ticket information, and assistance necessity information. The lost child data is a database created in response to a lost child report, and the characteristics of the lost child, the location where the child got lost, and the parent's contact information are registered in association with each other. The IC passenger ticket information is passenger information registered in association with the identification information of the IC passenger ticket. Here, the IC passenger ticket owned by the person requiring care registers the fact that the person requires care, the characteristics of the person requiring care such as a facial photo, and the necessary care. In addition, the presence of the person requiring care within the station can be identified, for example, by the ticket gate 14 reading the IC passenger ticket of the person requiring care. The assistance necessity information is information generated by a station staff member manually registering the same information for a person requiring care who does not have IC passenger ticket information registered.

[0075] The behavior or circumstances of people that are expected to be involved in responding to individuals requiring assistance include (a) lost children, (b) people requiring care, people who are unwell, or people who are intoxicated, (c) people in groups, and (d) people carrying heavy loads.

[0076] Regarding (a) above, the image recognition unit 22a can estimate that the person is a lost child, for example, based on the fact that the person has a troubled expression, that there is no other person who changes position with the person, that the person's age is less than a predetermined value, etc. Furthermore, the image recognition unit 22a refers to the lost child data, and when it identifies a person who matches the characteristics described in the lost child data, it estimates that there is a high possibility that the person is lost.

[0077] Regarding the above (b), the image recognition unit 22a can make the estimation by the above-mentioned example 3. Alternatively, the image recognition unit 22a can estimate that the target person is a person requiring care, a person in poor physical condition, or a person who is drunk, based on the IC ticket information, the information on whether assistance is required, the person's posture, and the person's movement.

[0078] Regarding the above (c), the image recognition unit 22a judges whether or not the people are acting in a group based on the image recognition information of a plurality of people. Specifically, the image recognition unit 22a estimates that the people are acting in a group when the number of people whose positions change together exceeds a predetermined number.

[0079] Regarding (d) above, the image recognition unit 22a estimates that the person is carrying heavy luggage based on the person's belongings. In addition, the image recognition unit 22a compares the person's position change with the coin locker information, and if the person's belongings do not change even though the person has moved from one coin locker to another, it additionally determines that the person is searching for a coin locker.

[0080] (a) In dealing with a lost child, for example, the robot 30 moves to the location of the lost child and calls out to the lost child. When it is determined that the child is lost, the robot 30 guides the child to the location of a station staff member. Instead of guiding the child to the location of the station staff member, the robot 30 may notify the station staff member of the location of the lost child. At this time, the station staff member may be additionally notified of the result of matching with lost child data. Furthermore, if the child has an IC passenger ticket, the robot 30 may obtain information about the child by reading the IC passenger ticket. This makes it possible to identify the child's name or the contact information of the parent, and this information may also be notified to the station staff member.

[0081] (b) In dealing with a person requiring care, a person in poor physical condition, or a person who is completely drunk, the robot 30 moves to the location of the person and provides the necessary assistance to the person. Assistance required for a person requiring care is, for example, assisting the person in moving to a place where the person cannot move alone. Assistance required for a person in poor physical condition or a person who is completely drunk is guiding the person to a rest area such as a bench, or notifying station staff. In addition, if the robot 30 cannot communicate with a person in poor physical condition or a person who is completely drunk, the robot 30 may obtain information about the person by reading their IC ticket in the same way as when a person is lost.

[0082] (c) When a person is in a group, the robot 30 moves to the location of the person and guides the group or monitors the person to prevent them from becoming separated.

[0083] (d) To deal with a person who has baggage that is difficult to carry, the robot 30 moves to the location of the person and carries the baggage on the person's behalf. Also, if the person is looking for a coin locker, the location of an empty coin locker may be presented based on the coin locker information.

[0084] By carrying out the above processing, the system can identify people who need attention and have the robot 30 deal with them. This can significantly reduce the burden on station staff compared to when station staff deal with the situation themselves. In addition, since the flowchart in FIG. 2 is performed on images of various locations within the station, maintenance of a wide area within the station can be performed by the system. This can also reduce the burden on station staff of having to continually watch footage from multiple surveillance cameras.

[0085] (Feature 1) The station premises maintenance system 1 of this embodiment includes an image recognition unit 22a, a command generation unit 22c, and a robot 30. The image recognition unit 22a sequentially acquires images generated by capturing images of the station premises with the imaging device 11 over time, and identifies information about people appearing in the images by image recognition. The command generation unit 22c determines how to deal with the person based on the information about the person identified by the image recognition unit 22a, and generates a command for the response. The robot 30 receives the command generated by the command generation unit 22c, moves to the position of the person based on the command, and deals with the person.

[0086] This allows any person in the station who needs some kind of action to be taken to respond using the robot 30. In particular, since the decision on the necessary action and the execution of the action are automated, the burden on station staff can be significantly reduced.

[0087] (Feature 2) In the station premises maintenance system 1 of this embodiment, the command generation unit 22c generates and transmits to the robot 30 a command to move the robot 30 to the position of a person, and a command to have the robot 30 speak to the person in response to the person, guide the person, or monitor the person.

[0088] This allows the system to recognize that there is a person who needs to be spoken to, guided, or monitored, and the robot 30 can be used to deal with the situation.

[0089] (Feature 3) In the station premises maintenance system 1 of this embodiment, the command generation unit 22c generates and transmits to the robot 30 a command to move the robot 30 to the position of a person, and a command to have the robot 30 guide the person to a position where a station staff member is waiting.

[0090] This allows people who need to be dealt with by station staff to be handed over to station staff, making it possible to deal with situations that the robot 30 alone cannot handle.

[0091] (Feature 4) In the station premises maintenance system 1 of this embodiment, the command generating unit 22c or the robot 30 has a function of controlling a facility for movement, which is a facility provided in the station premises and is used by a person for movement or a facility for restricting the movement of a person. The command generating unit 22c or the robot 30 controls the facility for movement based on information about the person.

[0092] This enables the system to stop people from moving or create new routes for them to move in the event of an emergency.

[0093] (Feature 5) In the station premises maintenance system 1 of this embodiment, the image recognition unit 22a performs image recognition on an image to determine whether a person appearing in the image is likely to collide with a railway vehicle, commit an act of violence, or commit sexual harassment in the present or future, and to identify any changes in the person's position.

[0094] This enables the system to automatically identify people within the station who require special attention.

[0095] (Feature 6) In the station premises maintenance system 1 of this embodiment, the image recognition unit 22a identifies, by image recognition of an image, that a person shown in the image is likely to be engaging in problematic behavior now or in the future, and identifies a change in the person's position. The command generation unit 22c generates a command to move the robot 30 to the position of the person who is likely to be engaging in problematic behavior, and a command to have the robot 30 speak to the person who is likely to be engaging in problematic behavior, and transmits these to the robot 30.

[0096] This allows the system to prevent problematic behavior before it occurs.

[0097] (Feature 7) In the station premises maintenance system 1 of this embodiment, the image recognition unit 22a performs image recognition on the image to determine that the person in the image is the person who has engaged in problematic behavior and to identify the change in the person's position. The command generation unit 22c generates a command to move the robot 30 to the position of the person who has engaged in problematic behavior and transmits the command to the robot 30. The command generation unit 22c or the robot 30 controls the transportation equipment around the person who has engaged in problematic behavior to restrict the movement route of the person who has engaged in problematic behavior.

[0098] This makes it possible to have the robot deal with the problematic behavior while preventing the individual from escaping.

[0099] (Feature 8) In the station premises maintenance system 1 of this embodiment, the command generation unit 22c generates and transmits to the robot 30 a command to move the robot 30 to the position of a person and a command to have the person present a detour route created based on railway vehicle operation information.

[0100] This allows the system to present passengers with detour routes created based on operational information.

[0101] (Feature 9) In the station premises maintenance system 1 of this embodiment, the image recognition unit 22a identifies the congestion state of each area in the station premises by image recognition of an image. The command generation unit 22c generates and transmits to the robot 30 a command to move the robot 30 to the crowded area and a command to have the robot 30 guide or organize people in the area.

[0102] This allows the system to identify the congestion situation and alleviate the congestion or reduce its impact.

[0103] (Feature 10) In the station premises maintenance system 1 of this embodiment, the image recognition unit 22a performs image recognition on the image to determine that the person shown in the image is a person requesting information regarding boarding a railroad car and to identify a change in the person's position. The command generation unit 22c generates and transmits to the robot 30 a command to move the robot 30 to the position of the person requesting information regarding boarding a railroad car and a command to have the robot 30 provide the person with information.

[0104] This enables the system to identify people who are having trouble boarding trains and take action.

[0105] (Feature 11) In the station premises maintenance system 1 of this embodiment, the image recognition unit 22a identifies that the person in the image is a lost child and identifies the change in the person's position by image recognition of the image. The command generation unit 22c generates a command to move the robot 30 to the location of the lost child and a command to have the robot 30 call out to the lost child, and transmits them to the robot 30.

[0106] This allows the system to identify lost children and take action.

[0107] (Feature 12) The station premises maintenance system 1 of this embodiment includes a plurality of robots 30 with different positions or functions. The command generation unit 22c determines the action and position to be executed by the robot 30, selects a robot 30 according to the action and position, and transmits a command to the selected robot 30.

[0108] This allows effective use of multiple robots 30 with different positions or functions.

[0109] (Feature 13) In the station premises maintenance system 1 of this embodiment, target person information based on the identification information of the IC ticket acquired by the ticket gate 14 is input to the command generation unit 22c. The command generation unit 22c generates a command to make the robot 30 take action according to the target person information, and transmits the command to the robot 30.

[0110] This allows the system to take appropriate action using the information from the IC card.

[0111] The above-mentioned features 1 to 13 can be combined, for example, as follows to realize the station premises maintenance system 1. The same applies to the station premises maintenance method or the station premises maintenance system program. [Configuration 1] Station premises maintenance system 1 with feature 1 [Configuration 2] Station premises maintenance system 1 with feature 2 in addition to configuration 1 [Configuration 3] Station premises maintenance system 1 having feature 3 in addition to configuration 1 or 2 [Configuration 4] Station premises maintenance system 1 having feature 4 in addition to any one of configurations 1 to 3 [Configuration 5] Station premises maintenance system 1 having feature 5 in addition to any one of configurations 1 to 4 [Configuration 6] In addition to any one of configurations 1 to 5, a station premises maintenance system 1 further having feature 6 [Configuration 7] Station premises maintenance system 1 having any one of configurations 1 to 6, and further having feature 7 [Configuration 8] In addition to any one of configurations 1 to 7, a station premises maintenance system 1 further having feature 8 [Configuration 9] In addition to any one of configurations 1 to 8, a station premises maintenance system 1 further having feature 9 [Configuration 10] In addition to any one of configurations 1 to 9, a station premises maintenance system 1 having a further feature 10 [Configuration 11] A station premises maintenance system 1 having any one of configurations 1 to 10, and further having feature 11. [Configuration 12] A station premises maintenance system 1 having any one of configurations 1 to 11 and further having feature 12. [Configuration 13] A station premises maintenance system 1 having any one of configurations 1 to 12 and further having feature 13.

[0112] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general purpose processors, special purpose processors, integrated circuits, Application Specific Integrated Circuits (ASICs), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. Processors are considered processing circuits or circuits because they include transistors and other circuits. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or the processor. [Explanation of symbols]

[0113] 1 Station premises maintenance system 22 Processing Server 22a Image recognition section 22b Analysis Department 22c Command generator 30. Robot

Claims

1. an image recognition unit that sequentially acquires images generated by capturing images of the inside of the station using an imaging device over time and identifies information about people appearing in the images through image recognition; a command generating unit that determines a countermeasure according to the person based on information of the person identified by the image recognition unit and generates a command for the countermeasure; a robot that receives the command generated by the command generation unit, moves to the position of the person based on the command, and deals with the person; This is an in-station maintenance system.

2. The station premises maintenance system according to claim 1, The command generation unit generates and transmits to the robot a command to move the robot to the position of the person and a command to have the robot speak to the person in response to the person, guide the person, or monitor the person.

3. The station premises maintenance system according to claim 1, The command generation unit generates and transmits to the robot a command to move the robot to the position of the person and a command to have the robot guide the person to a position where a station staff member is waiting.

4. The station premises maintenance system according to claim 1, the command generating unit or the robot has a function of controlling a facility for movement that is provided in a station and is used by the person for movement or that regulates the movement of the person, A station premises maintenance system, wherein the command generating unit or the robot controls the mobile equipment based on information about the person.

5. The station premises maintenance system according to claim 1, The image recognition unit performs image recognition on the image to determine whether the person shown in the image is likely to collide with a railway vehicle, commit an act of violence, or commit sexual harassment in the present or future, and to identify any changes in the position of the person.

6. The station premises maintenance system according to claim 1, the image recognition unit, by image recognition of the image, identifies that the person shown in the image is likely to currently or in the future engage in problematic behavior and identifies a change in position of the person; The command generation unit generates and transmits to the robot a command to move the robot to the location of a person who may be engaging in problematic behavior, and a command to have the robot speak to the person who may be engaging in problematic behavior.

7. The station premises maintenance system according to claim 4, The image recognition unit identifies, by image recognition of the image, that the person appearing in the image has engaged in problematic behavior and identifies a positional change of the person; the command generation unit generates a command to move the robot to a position of a person who has engaged in problematic behavior, and transmits the command to the robot; A station premises maintenance system, wherein the command generation unit or the robot controls the mobility equipment around the person who has engaged in problematic behavior to restrict the movement route of the person who has engaged in problematic behavior.

8. The station premises maintenance system according to claim 1, The command generation unit generates and transmits to the robot a command to move the robot to the position of the person and a command to have the person present a detour route created based on railway vehicle operation information.

9. The station premises maintenance system according to claim 1, The image recognition unit identifies a congestion state for each area in a station by image recognition of the image, The command generation unit generates and transmits to the robot a command to move the robot to an area where congestion is occurring and a command to have the robot guide or organize the people in the area.

10. The station premises maintenance system according to claim 1, the image recognition unit, by image recognition of the image, determines that the person shown in the image is a person requesting information regarding riding in a railway vehicle and identifies a change in position of the person; The command generation unit generates and transmits to the robot a command to move the robot to the location of a person requesting information regarding boarding a railway vehicle, and a command to have the robot provide information to the person.

11. The station premises maintenance system according to claim 1, the image recognition unit identifies, by image recognition of the image, that the person shown in the image is a lost child and identifies a change in position of the person; The command generating unit generates and transmits to the robot a command to move the robot to the location of a lost child and a command to have the robot call out to the lost child.

12. The station premises maintenance system according to claim 1, A plurality of the robots having different positions or functions are provided, The command generation unit determines an action and a location to be executed by the robot, selects the robot according to the action and the location, and transmits the command to the selected robot.

13. The station premises maintenance system according to claim 1, The command generating unit receives target person information based on the identification information of the IC ticket acquired by the ticket gate, The command generation unit generates a command to cause the robot to take action according to the target person information and transmits the command to the robot.

14. Images of the inside of a station taken by an imaging device are sequentially acquired over time, and information on people appearing in the images is identified by image recognition; determining a countermeasure for the person based on information about the person identified by the image recognition, generating a command for the countermeasure and transmitting the command to the robot; The station premises maintenance method includes causing the robot, upon receiving the command, to move to the position of the person based on the command and deal with the person.

15. Images of the inside of a station taken by an imaging device are sequentially acquired over time, and information on people appearing in the images is identified by image recognition; determining a countermeasure for the person based on the information of the person identified by the image recognition, generating a command for the countermeasure and transmitting the command to the robot; A station maintenance program that causes a computer to execute processes.

Images