Information processing device and information processing system

Abstract

[Problem] To automatically identify work to be performed on a vehicle. [Solution] An information processing device comprising a processing unit that identifies the start of work on the basis of the overlapping of a first detection target of a vehicle and a second detection target that performs work on the vehicle in an image captured by a camera.

Description

Information Processing Apparatus and Information Processing System

[0001] The present disclosure relates to an information processing apparatus and an information processing system.

[0002] In recent years, technologies for recognizing the state of vehicles have been demanded. For example, Patent Document 1 below discloses a technology for recognizing the type of a vehicle based on a captured image of the vehicle's license plate obtained by a camera.

[0003] Japanese Unexamined Patent Application Publication No. 2020-135535

[0004] On the other hand, in industrial fields that use vehicles such as logistics, retail, delivery, or transportation, it has been demanded to grasp more details of the operations performed on the vehicles in order to improve the efficiency of operations.

[0005] Therefore, the present disclosure proposes a novel and improved information processing apparatus and an information processing system that can automatically recognize the operations performed on a vehicle.

[0006] According to the present disclosure, there is provided an information processing apparatus including a processing unit that recognizes the start of the operation based on the overlap between a first detection target of a vehicle in an image obtained by a camera and a second detection target that performs an operation on the vehicle.

[0007] Further, according to the present disclosure, there is provided an information processing system including a processing unit that recognizes the start of the operation based on the overlap between a first detection target of a vehicle in an image obtained by a camera and a second detection target that performs an operation on the vehicle, an output unit that outputs the recognized start time of the operation, and a management unit that manages the output start time of the operation.

[0008] This is an explanatory diagram illustrating the overall configuration of an information processing system according to one embodiment of the present disclosure. This is a schematic diagram showing the positional relationship between a camera and a parking space where a vehicle is parked. This is a block diagram showing an example of the configuration of a camera included in the information processing system. This is a perspective view showing an example of the external configuration of a camera. This is a block diagram showing the functional configuration of a camera included in the information processing system. This is an explanatory diagram illustrating the case when the camera recognizes the start of work on a vehicle. This is an explanatory diagram illustrating an example when the camera recognizes the end of work on a vehicle. This is an explanatory diagram illustrating another example when the camera recognizes the end of work on a vehicle. This is a block diagram illustrating the functional configuration of a management server included in the information processing system. This is a table diagram showing an example of management information stored in the management server. This is a sequence diagram showing the operation flow of the information processing system. This is a block diagram showing an example of the hardware configuration of an information processing device that realizes information processing by the camera and the management server.

[0009] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] The explanation will be given in the following order: 1. Overall configuration of the information processing system 2. Configuration of each part of the information processing system 2.1. Camera configuration 2.2. Management server configuration 3. Operation of the information processing system 4. Example of hardware configuration

[0011] <1. Overall Configuration of the Information Processing System> First, the overall configuration of the information processing system according to one embodiment of the present disclosure will be described with reference to Figures 1 and 2. Figure 1 is an explanatory diagram illustrating the overall configuration of the information processing system 1 according to this embodiment. Figure 2 is a schematic diagram showing the positional relationship between the camera 10 and the parking space PA where the vehicle TK is parked.

[0012] As shown in Figure 1, the information processing system 1 according to this embodiment includes a camera 10 and a management server 20.

[0013] Camera 10 is an imaging device positioned facing the parking space PA where the vehicle TK is parked. The parking space PA is a space for performing work on the parked vehicle TK, and camera 10 can image the vehicle TK parked in the parking space PA and the work performed on the vehicle TK. For example, the parking space PA may be a berth provided at each location in industrial fields such as logistics, retail, distribution, or transportation. At a berth, loading and unloading of goods is performed on transport trucks parked at the berth. In other words, the work performed on the vehicle TK parked in the parking space PA is the loading and unloading of goods onto the cargo bed of a transport truck.

[0014] In this specification, loading and unloading of goods onto a truck bed means both loading goods onto the truck bed and unloading goods from the truck bed. Loading and unloading of goods may occur simultaneously with unloading from the truck bed and loading goods onto the truck bed.

[0015] Furthermore, the camera 10 includes a recognition device that recognizes the object to be detected from the captured image. As a result, the camera 10 can recognize the object to be detected from the captured image and output the time at which the object was recognized, as well as an image of the object extracted from the captured image. The time at which the object was recognized and the image of the object extracted from the object, output from the camera 10, are transmitted to, for example, the management server 20. Therefore, in this embodiment, the information processing system 1 recognizes the work performed on the vehicle TK parked in the parking space PA using the camera 10, making it possible for the management server 20 to determine the start time of the work performed on the vehicle TK.

[0016] The management server 20 is an information processing server that manages the use of parking spaces PA by vehicles TK. Specifically, the management server 20 may manage the time when vehicles TK use parking spaces PA. For example, the management server 20 may manage the entry and exit times of vehicles TK to the location where parking spaces PA are provided, the entry and exit times of vehicles TK to parking spaces PA, and the start and end times of work performed on vehicles TK parked in parking spaces PA. This makes it possible for the management server 20 to manage the progress of work performed on vehicles TK parked in parking spaces PA.

[0017] Camera 10 and parking space PA are pre-associated by the management server 20. For example, the management server 20 may recognize the image transmitted from camera 10 as an image of the parking space PA associated with camera 10, based on correspondence information indicating the correspondence between camera 10 and parking space PA.

[0018] The camera 10 and the management server 20 are connected via network N. The camera 10 and the management server 20 may be located at different locations or at the same location. Network N is a network connected by wire or wireless. Network N may be, for example, an Internet communication network, a mobile communication network, a local area network (LAN), a wide-area Ethernet, an infrared communication network, a radio wave communication network, or a satellite communication network.

[0019] Camera 10 may be installed, for example, on the ceiling at the rear of the parking space PA. In this case, camera 10 can capture an image of the rear of a vehicle TK parked in the parking space PA. The positional relationship between camera 10 and the parking space PA where the vehicle TK is parked will be explained in more detail with reference to Figure 2.

[0020] As shown in Figure 2, multiple parking spaces PA, which are loading and unloading berths, are provided at each location in industrial fields such as logistics, retail, distribution, or transportation. Vehicles TK enter the parking space PA from the rear side, where a door (rear door) leading to the cargo bed is provided, and park in the parking space PA. This allows workers to load and unload cargo from the rear door of the vehicle TK. The parking space PA may be a high-floor berth with a raised floor section at the rear of the parking space PA that is approximately the same height as the cargo bed of the vehicle TK, or it may be a low-floor berth with a floor that is the same height as the ground.

[0021] Camera 10 is installed on the ceiling at the rear of the parking space PA, allowing it to capture images of workers loading and unloading cargo through the rear door opening without interfering with their work. Furthermore, camera 10 can capture an image of the entire vehicle TK and the license plate attached to the rear of the vehicle TK from the rear of the parking space PA.

[0022] Furthermore, if the vehicle TK is equipped with wing side panels that allow the cargo bed to be opened from the side, the camera 10 may be installed on the side of the vehicle TK parked in the parking space PA so as to be able to capture images of the opening to the cargo bed by the wing side panels.

[0023] The information processing system 1 according to this embodiment recognizes the work performed on the vehicle TK using the camera 10 and outputs the start time of the work based on the recognition result, thereby enabling more efficient management of the work performed on the vehicle TK.

[0024] In the following, the configurations of the camera 10 and the management server 20 included in the information processing system 1 according to this embodiment will be described in more detail.

[0025] <2. Configuration of each part of the information processing system> (2.1. Camera configuration) Referring to Figures 3 to 8, the configuration of the camera 10 included in the information processing system 1 according to this embodiment will be described.

[0026] Figure 3 is a block diagram showing an example configuration of a camera 10 included in the information processing system 1. As shown in Figure 3, the camera 10 includes an imaging block 20 and a signal processing block 30. The imaging block 20 and the signal processing block 30 are connected to each other via internal buses, namely connecting lines CL1, CL2, and CL3, enabling data exchange between them.

[0027] The imaging block 20 is a functional block that performs imaging functions and includes an imaging unit 21, an imaging processing unit 22, an output control unit 23, an output I / F 24, and an imaging control unit 25.

[0028] The imaging unit 21 is composed of multiple pixels arranged in a two-dimensional array. The imaging unit 21 is driven by the imaging processing unit 22 to capture an image. Specifically, the imaging unit 21 receives light incident through the optical system at each pixel and generates an analog image signal corresponding to the light incident on the camera 10 by converting the received light into photoelectricity. The type of image generated by the imaging unit 21 may be, for example, an RGB (Red, Green, Blue) color image or a monochrome image containing only luminance information.

[0029] The imaging processing unit 22 performs processing related to image acquisition by the imaging unit 21 in accordance with the control of the imaging control unit 25. For example, the imaging processing unit 22 may perform driving of the imaging unit 21, AD (Analog to Digital) conversion of the analog image signal generated by the imaging unit 21, and imaging signal processing. Imaging signal processing includes, for example, processing to calculate the brightness of each small area by calculating the average value of the pixel values ​​for each predetermined small area, processing to convert the image generated by the imaging unit 21 into an HDR (High Dynamic Range) image, defect correction processing, or development processing.

[0030] The imaging processing unit 22 generates a digital image signal by performing AD conversion on the analog image signal generated by the imaging unit 21. The generated digital image signal is output to the output control unit 23 and also to the image compression unit 35 of the signal processing block 30 via the connection line CL2.

[0031] The output control unit 23 controls the selective output to the outside from the output I / F 24. Specifically, the output control unit 23 may control the output to the outside of the image signal output from the imaging processing unit 22, or it may control the output to the outside of the recognition result of the captured image performed by the signal processing block 30, which will be described later.

[0032] The output I / F 24 is an interface that outputs various information to the outside according to the control of the output control unit 23. For example, the output I / F 24 may output the image signal supplied from the imaging processing unit 22 and the recognition result of the captured image performed by the signal processing block 30, which will be described later, to the outside.

[0033] For example, if only the recognition result of the captured image (so-called metadata) is required from the camera 10, the output control unit 23 may control the output I / F 24 to output only the recognition result of the captured image to the outside. In such a case, the output control unit 23 can reduce the amount of data output to the outside from the output I / F 24, making it possible to output data to the outside at a faster speed.

[0034] The imaging control unit 25 controls the imaging processing unit 22 according to the imaging information stored in the register group 27, thereby controlling the image acquisition by the imaging unit 21.

[0035] The register group 27 includes a plurality of registers that store setting information related to imaging by the imaging unit 21. For example, the register group 27 may store setting information related to imaging received from an external source via the communication I / F 26, or the results of imaging signal processing by the imaging processing unit 22. Examples of setting information related to imaging include setting information such as analog gain, exposure, shutter speed, frame rate, focus, white balance, or crop range.

[0036] Furthermore, the register group 27 may store additional setting information related to output control in the output control unit 23. The output control unit 23 can selectively control output to the outside according to the setting information related to output control.

[0037] The communication interface 26 is an interface for sending and receiving data with the outside. The communication interface 26 may, for example, send and receive various setting information stored in the register group 27 with the outside.

[0038] The signal processing block 30 is a functional block that performs a recognition function to recognize captured images, and includes a CPU (Central Processing Unit) 31, a DSP (Digital Signal Processor) 32, a memory 33, a communication interface 34, an image compression unit 35, and an input interface 36. The CPU 31, DSP 32, memory 33, communication interface 34, image compression unit 35, and input interface 36 are connected via a bus, allowing them to input and output information to each other.

[0039] The CPU 31 controls the signal processing block 30 and inputs and outputs information to the imaging control unit 25 via the connecting line CL1 by executing a program stored in the memory 33.

[0040] For example, the CPU 31 may instruct the imaging control unit 25 to perform imaging via the connection line CL1. Alternatively, the CPU 31 may instruct the imaging control unit 25 to perform imaging based on the settings information written to the register group 27 of the imaging control unit 25.

[0041] The DSP 32 executes AI processing on the captured image supplied from the imaging processing unit 22 via the connection line CL2 by executing a program stored in the memory 33. The content of the AI ​​processing executed by the DSP 32 is determined by the content of the AI ​​model held in the DSP 32. The DSP 32 can switch the content of the AI ​​processing to be executed by receiving various AI models from an external source.

[0042] AI models may be generated by machine learning algorithms such as DNN (Deep Neural Network), CNN (Convolutional Neural Network), or Transformer. Alternatively, AI models may be generated by rule-based algorithms.

[0043] The memory 33 stores data and the like used in the processing of the signal processing block 30. The memory 33 may be composed of an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), or the like. For example, the memory 33 may store a program received from the outside via the communication I / F 34, an imaging image output from the imaging block 20, the result of AI processing executed by the DSP 32, and setting information received from the input I / F 36.

[0044] The communication I / F 34 is an interface for transmitting and receiving data to and from the outside. The communication I / F 34 may transmit and receive information such as a program executed by the CPU 31 or the DSP 32 to and from the outside. For example, the communication I / F 34 may store a program downloaded from the outside in the memory 33 for execution by the CPU 31 or the DSP 32.

[0045] The image compression unit 35 performs compression processing on the imaging image transferred via the connection line CL2 from the imaging processing unit 22. Thereby, the image compression unit 35 can generate a compressed image having a smaller data amount than the imaging image before compression.

[0046] The AI processing in the DSP 32 may be performed on either the imaging image captured by the imaging block 20 or the compressed image generated by the image compression unit 35. Since the compressed image has a smaller data amount than the imaging image, the load of the AI processing in the DSP 32 can be reduced, and the storage capacity of the memory 33 for storing the compressed image can be saved. As the compression processing in the image compression unit 35, for example, compression processing for scaling down an imaging image of 12M (3968×2976) pixels to an image of VGA (640×480) size can be exemplified.

[0047] The input I / F 36 is an interface that receives various information from the outside (e.g., an external sensor). For example, the input I / F 36 may receive the sensing result obtained by an external sensor. The received sensing result may be, for example, AI-processed by the DSP 32. Examples of the external sensor include a distance measurement sensor that acquires depth information, an infrared sensor that acquires an infrared image, or an image sensor different from the camera 10.

[0048] The camera 10 having the above configuration can perform AI processing on the captured image captured by the imaging unit 21. As a result, the camera 10 can output the recognition result by AI processing from the output I / F 24 separately from the captured image.

[0049] FIG. 4 is a perspective view showing a configuration example of the appearance of the camera 10. As shown in FIG. 4, the camera 10 is configured as a semiconductor device having a stacked structure in which a first substrate 51 and a second substrate 52 are stacked.

[0050] For example, the imaging unit 21 is provided on the first substrate 51. On the other hand, for example, the imaging processing unit 22, the output control unit 23, the output I / F 24, the imaging control unit 25, the CPU 31, the DSP 32, the memory 33, the communication I / F 34, the image compression unit 35, and the input I / F 36 are provided on the second substrate 52.

[0051] As an example, the first substrate 51 and the second substrate 52 may be electrically connected by a through-via that penetrates the first substrate 51 and reaches the second substrate 52. Alternatively, as another example, the first substrate 51 and the second substrate 52 may be electrically connected by a Cu-Cu bonding that directly bonds the Cu wiring exposed on the lower surface side of the first substrate 51 and the Cu wiring exposed on the upper surface side of the second substrate.

[0052] If the area of the camera 10 is allowed to be large, the camera 10 may be composed of a single substrate.

[0053] Furthermore, the camera 10 may be constructed by stacking three or more substrates. In such a case, the camera 10 may be constructed by stacking a first substrate on which an imaging unit 21 is provided, a second substrate on which an imaging processing unit 22, an output control unit 23, an output I / F 24, an imaging control unit 25, a CPU 31, a DSP 32, a communication I / F 34, an image compression unit 35, and an input I / F 36, and a third substrate on which a memory 33 is provided.

[0054] Figure 5 is a block diagram showing the functional configuration of the camera 10 included in the information processing system 1. Figure 6 is an explanatory diagram illustrating the case when the camera 10 recognizes the start of work on the vehicle TK. Figure 7 is an explanatory diagram illustrating an example of the case when the camera 10 recognizes the end of work on the vehicle TK. Figure 8 is an explanatory diagram illustrating another example of the case when the camera 10 recognizes the end of work on the vehicle TK.

[0055] As shown in Figure 5, the camera 10 included in the information processing system 1 comprises an image acquisition unit 110, an ISP unit 120, a processing unit 130, and an output unit 140.

[0056] The image acquisition unit 110 corresponds to the imaging unit 21 described above and acquires an image signal that reflects an image in real space. Specifically, the image acquisition unit 110 acquires an image signal by converting light captured from real space into an electrical signal using an image sensor. For example, the image acquisition unit 110 may convert light into an electrical signal using a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0057] The ISP unit 120 corresponds to the imaging processing unit 22 described above and performs various image processing on the image signal acquired by the image acquisition unit 110. Specifically, the ISP unit 120 may perform various image processing on the image signal, such as demosaicing, sharpening, noise reduction, resolution conversion, digital gain adjustment, tone mapping, color correction, color conversion, normalization, or quantization. The ISP unit 120 may be, for example, an Image Signal Processor.

[0058] The processing unit 130 uses a model generated through learning to recognize the object to be detected from the image processed by the ISP unit 120. The model used for recognition may consist of, for example, a DNN (Deep Neural Network) or a CNN (Convolutional Neural Network). The processing unit 130 further generates metadata using the recognized object to be detected.

[0059] For example, the processing unit 130 may recognize the start of work on a vehicle TK parked in a parking space PA based on the recognition result of the detected object in the image. Specifically, the processing unit 130 may recognize the start of work on a vehicle TK by recognizing the opening OD of the rear door of the vehicle TK and the worker WR or work machine performing work on the vehicle TK from the image.

[0060] As shown in Figure 6, first, the processing unit 130 recognizes the opening OD of the rear door leading to the cargo bed of the vehicle TK as the first detection target from the image. Next, the processing unit 130 sets a bounding box by enclosing the recognized opening OD with a minimum rectangle.

[0061] On the other hand, the processing unit 130 recognizes a worker WR performing work on the vehicle TK, or a work machine operated by the worker, as a second detection target from the image. Examples of work machines operated by workers include forklifts, carriers, carts, turrets, or trolleys. Subsequently, the processing unit 130 sets a bounding box, which is a partial region enclosed by the smallest rectangle around the recognized worker WR or work machine.

[0062] Subsequently, the processing unit 130 recognizes the start of work on the vehicle TK based on the overlap between the opening OD, which is the first detection target, and the worker WR or work machine, which is the second detection target. Specifically, the processing unit 130 may recognize the start of work on the vehicle TK based on at least one of the following: the overlapping area between the bounding box of the opening OD and the bounding box of the worker WR or work machine, or the elapsed time since the overlap occurred. For example, the processing unit 130 may recognize the start of work on the vehicle TK when the overlapping area between the bounding box of the opening OD and the bounding box of the worker WR or work machine exceeds a threshold, when the elapsed time since the overlap occurred exceeds a threshold, or when both of the above two conditions are met.

[0063] During the loading and unloading of cargo onto the cargo bed of vehicle TK, the worker WR or the work machine will pass through the rear door opening OD to enter the cargo bed of vehicle TK. Therefore, the processing unit 130 can recognize that the loading and unloading of cargo onto the cargo bed of vehicle TK has begun by recognizing the overlap between the opening OD and the worker WR or work machine. Furthermore, by setting thresholds for the area or time of overlap between the opening OD and the worker WR or work machine, the processing unit 130 can prevent misrecognition of the start of work, such as when the worker WR passes in front of the rear door of vehicle TK without performing any work.

[0064] For example, the processing unit 130 may recognize the completion of work on the vehicle TK parked in the parking space PA based on the recognition result of the detected object in the image. Specifically, the processing unit 130 may recognize the completion of work on the vehicle TK when the opening OD, which is the first detected object, is no longer recognized in the image.

[0065] As shown in Figure 7, the processing unit 130 may recognize the completion of work on the vehicle TK when the opening OD (first detection target) that has formed on the rear of the vehicle TK is closed by the rear door RD and is no longer recognized in the image. Alternatively, the processing unit 130 may recognize the completion of work on the vehicle TK when the rear door RD that closes the opening OD leading to the cargo bed of the vehicle TK is recognized as a detection target in the image. In loading and unloading operations on the vehicle TK, loading or unloading is performed on the cargo bed of the vehicle TK. Therefore, the processing unit 130 can recognize the completion of loading and unloading operations on the cargo bed by recognizing that the opening OD leading to the cargo bed is closed by the rear door RD.

[0066] Alternatively, the processing unit 130 may recognize the completion of work on the vehicle TK when the size of the vehicle TK in the image becomes smaller and it is recognized that the vehicle TK has left the parking space PA.

[0067] As shown in Figure 8, the processing unit 130 may recognize the completion of work on the vehicle TK if it recognizes that the vehicle TK has left the parking space PA based on the size of the vehicle TK's body or the size of the vehicle TK's license plate in the image. For example, the processing unit 130 recognizes the vehicle body TW or license plate NP of the vehicle TK as a detection target from the image and sets a bounding box by enclosing the recognized vehicle body TW or license plate NP with the smallest possible rectangle. Subsequently, the processing unit 130 may recognize that the vehicle TK has left the parking space PA and recognize the completion of work on the vehicle TK if the size of the bounding box of the vehicle body TW or license plate NP becomes smaller than or equal to the respective threshold.

[0068] During loading and unloading operations for vehicle TK, vehicle TK may exit the parking space PA with the opening OD leading to the cargo bed open. In such cases, the driver of vehicle TK will close the opening OD leading to the cargo bed by closing the rear door RD after exiting the parking space PA but before leaving the base. Therefore, the processing unit 130 can recognize the completion of the work on vehicle TK by recognizing that vehicle TK has exited the parking space PA.

[0069] As described above, the processing unit 130 can recognize the start of work on the vehicle TK parked in the parking space PA based on the recognition result of the detected object from the image. Therefore, the processing unit 130 can recognize the start time of the work performed on the vehicle TK by referring to the clock built into the camera 10. Furthermore, the processing unit 130 can recognize the end of the work performed on the vehicle TK parked in the parking space PA based on the recognition result of the detected object from the image. Therefore, similarly, the processing unit 130 can recognize the end time of the work performed on the vehicle TK by referring to the clock built into the camera 10.

[0070] More specifically, the processing unit 130 includes a model processing unit and a post-processing unit. The model processing unit performs the above-described processes, including detection of the rear door opening OD of the vehicle TK, the rear door RD that closes the opening OD, the worker WR, or the work machine, and detection of the position and size of the bounding box of the detected object, using the model generated by learning. The processing of the model processing unit may be performed by the DSP 32. The post-processing unit recognizes the start or end of work on the vehicle TK based on the overlap or size of the bounding boxes of the detected object. The processing of the post-processing unit may be performed by the CPU 31.

[0071] However, the model processing and post-processing may be performed together by the DSP 32 or by the CPU 31. Furthermore, the camera 10 may transmit the output of the model processing unit, performed by the DSP 32 or CPU 31, to an external server via the network N, thereby allowing the external server to perform the post-processing.

[0072] The output unit 140 corresponds to the output I / F 24 described above and outputs to the outside the image of the detection target recognized by the processing unit 130, and the metadata generated using the recognized detection target.

[0073] Specifically, the output unit 140 may output to the outside the start time of the work performed on the vehicle TK and the cropped image of the detection target used to recognize the start time of the work. The detection target used to recognize the start time of the work may be, for example, the opening OD of the rear door leading to the cargo bed of the vehicle TK, and the worker WR or work machine performing the work on the vehicle TK.

[0074] Furthermore, the output unit 140 may output to the outside the completion time of the work performed on the vehicle TK and the cropped image of the detection target used to recognize the completion time of the work. The detection target used to recognize the completion time of the work may be, for example, the rear door RD that closes the opening OD leading to the cargo bed of the vehicle TK, the vehicle body TW of the vehicle TK, or the license plate NP of the vehicle TK.

[0075] The start and end times of the work output from the output unit 140 are transmitted to the management server 20, for example, via the network N. Therefore, the management server 20 can grasp the start and end times of the work performed on the vehicle TK parked in the parking space PA, making it possible to manage the work performed in the parking space PA more efficiently.

[0076] On the other hand, the various cropped images output from the output unit 140 may be transmitted, for example, to a training server (not shown) via the network N. The training server is a server that collects cropped images of various detection targets recognized by the camera 10. The training server can assign correct labels to the collected cropped images, thereby allowing the model used for retraining the model to recognize the detection targets to use the cropped images with correct labels. In this way, the training server can further improve the accuracy of the model used for retraining the model to recognize the detection targets by having the model retrained with images of the detection targets actually captured by the camera 10.

[0077] The camera 10 included in the information processing system 1 according to this embodiment can recognize the start of work performed on a vehicle TK parked in a parking space PA and output the start time of the recognized work. Furthermore, the camera 10 can recognize the end of work performed on a vehicle TK parked in a parking space PA and output the end time of the recognized work. Therefore, since the information processing system 1 according to this embodiment can automatically recognize the start and end of work performed on a vehicle TK parked in a parking space PA, it is possible to manage the use of the parking space PA by the vehicle TK more efficiently.

[0078] (2.2. Configuration of the Management Server) The configuration of the management server 20 included in the information processing system 1 according to this embodiment will be described with reference to Figures 9 and 10. Figure 9 is a block diagram showing the functional configuration of the management server 20 included in the information processing system 1. Figure 10 is a table diagram showing an example of management information stored in the management server 20.

[0079] As shown in Figure 9, the management server 20 included in the information processing system 1 comprises a communication unit 210, a management unit 220, and a management information storage unit 230.

[0080] The communication unit 210 is a communication interface, for example, composed of a communication device for connecting to a network N. The communication unit 210 receives, for example, the start time of an operation performed on a vehicle TK parked in a parking space PA, which is output from the output unit 140 of the camera 10. The communication unit 210 may further receive the end time of the operation. The communication unit 210 may be, for example, a communication card for wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or WUSB (Wireless USB). Alternatively, the communication unit 210 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0081] The management unit 220 manages the use of the parking space PA by the vehicle TK. Specifically, the management unit 220 may manage the start time of the work performed on the vehicle TK output from the camera 10. The management unit 220 may further manage the end time of the work along with the start time. For example, the management unit 220 may manage the start and end times of the work performed on the vehicle TK as management information in association with the vehicle number of the vehicle TK. In this way, the management unit 220 can manage the progress of the work performed on the vehicle TK parked in the parking space PA.

[0082] As shown in Figure 10, the management unit 220 may manage the following as management information: the time of entry (gate entry) and exit (gate exit) of a vehicle TK to a base where a parking space PA is provided, the time of entry (berth entry) and exit (berth exit) of a vehicle TK to a parking space PA, and the time of start (loading start) and end (loading end) of work on a vehicle TK parked in a parking space PA. In other words, the management unit 220 can manage the start and end times of each event from the time a vehicle TK enters a base where a parking space PA is provided until it leaves, in association with the vehicle number of the vehicle TK and the parking space PA number (berth number).

[0083] For example, the management information shown in Figure 10 indicates that vehicle TK with vehicle number "TKP100 KA1234" is using parking space PA with berth number "101" and that work is being performed on vehicle TK. The management information also indicates the start and end times for each event, showing that vehicle TK entered the gate at the base where parking space PA is located at "10:00", vehicle TK entered the berth at parking space PA at "10:45", the start time of loading and unloading when work on vehicle TK began at "11:30", the end time of loading and unloading when work on vehicle TK was completed at "12:05", vehicle TK exited the berth from parking space PA at "12:20", and vehicle TK left the gate at the base where parking space PA is located at "12:30".

[0084] The management unit 220 manages the management information described above, enabling it to execute alert processing when delays occur in the progress of work on the vehicle TK. Alert processing is, for example, a process that notifies the administrator of the management server 20 or the driver of the vehicle TK that the progress of the work is delayed. Notification of delays in the progress of the work may also be made by sending a short message or the like to a terminal held by the administrator of the management server 20 or the driver of the vehicle TK.

[0085] Specifically, the management unit 220 may further calculate the work time taken for the work performed on the vehicle TK from the start and end times of the work, and execute an alert based on the calculated work time. For example, if the calculated work time is above a threshold, the management unit 220 may determine that the progress of the work performed on the vehicle TK is behind schedule and execute an alert.

[0086] Furthermore, if the start and end times for work on the vehicle TK are reserved for each parking space PA, the management unit 220 may perform alert processing based on the difference between the reserved start and end times for the work on the vehicle TK and the actual start and end times for the work on the vehicle TK. For example, if the difference between the reserved start and end times and the actual start and end times is greater than or equal to a threshold, the management unit 220 may determine that the progress of the work performed on the vehicle TK is behind schedule and perform alert processing.

[0087] The management information storage unit 230 is a data storage device configured as an example of a storage unit of the management server 20. The management information storage unit 230 may be composed of, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The management information storage unit 230 may store, for example, the management information shown in Figure 10.

[0088] The management server 20 included in the information processing system 1 according to this embodiment can manage the start and end times of work on the vehicle TK output from the camera 10 in association with the vehicle number of the vehicle TK and the berth number of the parking space PA. As a result, the management server 20 can more accurately grasp the progress of work on the vehicle TK parked in the parking space PA, and thus it is possible to manage the use of the parking space PA by the vehicle TK.

[0089] <3. Operation of the Information Processing System> Next, an example of the operation of the information processing system 1 according to this embodiment will be described with reference to Figure 11. Figure 11 is a sequence diagram showing the flow of operation of the information processing system 1.

[0090] As shown in Figure 11, first, the vehicle TK enters a parking space PA provided at each location (S101). At this time, the camera 10 installed in the parking space PA acquires an image of the vehicle TK parked in the parking space PA (S103), and recognizes the vehicle TK's entry from the acquired image (S105). The camera 10 outputs the time when it recognized the vehicle TK's entry as the vehicle TK's entry time to the management server 20 (S107).

[0091] Next, loading and unloading operations begin on the vehicle TK in the parking space PA (S111). At this time, camera 10 acquires images of the vehicle TK parked in the parking space PA and the worker WR performing operations on the vehicle TK (S113), and recognizes the start of loading and unloading operations from the acquired images (S115). For example, camera 10 may recognize the start of loading and unloading operations based on the overlap between the opening OD of the vehicle TK's cargo bed and the worker WR in the acquired images. Camera 10 outputs the time at which it recognized the start of loading and unloading operations as the start time of loading and unloading operations to the management server 20 (S117).

[0092] Next, the loading and unloading operation on the vehicle TK is completed (S121). At this time, camera 10 acquires an image of the vehicle TK parked in the parking space PA (S123), and recognizes the completion of the loading and unloading operation from the acquired image (S125). For example, camera 10 may recognize the completion of the loading and unloading operation when the opening RO to the loading bed of the vehicle TK is no longer recognized in the acquired image. Alternatively, camera 10 may recognize the completion of the loading and unloading operation when the size of the vehicle TK or license plate NP in the acquired image becomes smaller than or equal to the respective threshold. Camera 10 outputs the time at which it recognized the completion of the loading and unloading operation as the completion time of the loading and unloading operation to the management server 20 (S127).

[0093] Subsequently, vehicle TK exits the parking space PA (S131). At this time, camera 10 installed in the parking space PA acquires an image of the parking space PA (S133) and recognizes vehicle TK's exit from the acquired image (S135). Camera 10 outputs the time when it recognized vehicle TK's exit as the vehicle TK's exit time to the management server 20 (S137).

[0094] Furthermore, the management server 20 manages the vehicle TK's entry and exit times from the parking space PA, the start and end times of work performed on the vehicle TK, and the vehicle TK's license plate number as management information by associating them with each other (S149). As a result, the management server 20 can more accurately grasp the progress of work performed on the vehicle TK parked in the parking space PA, and thus manage the vehicle TK's use of the parking space PA.

[0095] As shown in the above example of operation, the information processing system 1 can automatically recognize the start and end of work performed on a vehicle TK parked in a parking space PA. Furthermore, by managing the start and end times of work performed on vehicle TK as management information in association with the vehicle's license plate number, the information processing system 1 can more efficiently manage the use of the parking space PA by vehicle TK.

[0096] <4. Hardware Configuration Example> Next, with reference to Figure 12, a hardware configuration for realizing information processing by the camera 10 and management server 20 included in the information processing system 1 according to this embodiment will be described. Figure 12 is a block diagram showing an example of the hardware configuration of the information processing device 900 that realizes information processing by the camera 10 and management server 20.

[0097] The functions of the information processing device 900 according to this embodiment can be realized through the cooperation of software and the hardware described below. The functions of the processing unit 130 of the camera 10 and the management unit 220 of the management server 20 may be executed by, for example, the CPU 901.

[0098] As shown in Figure 12, the information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903.

[0099] Furthermore, the information processing device 900 may further include a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 910, or a communication device 911. In addition, the information processing device 900 may have a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) in place of, or together with, the CPU 901.

[0100] The CPU 901 functions as an arithmetic processing unit or control unit, and controls the operation within the information processing unit 900 according to various programs recorded on the ROM 902, RAM 903, storage device 908, or removable recording medium installed in the drive 909. The ROM 902 stores programs used by the CPU 901, as well as arithmetic parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, as well as parameters used during its execution.

[0101] The CPU 901, ROM 902, and RAM 903 are interconnected by a host bus 904a capable of high-speed data transmission. The host bus 904a is connected to an external bus 904b, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 904, and the external bus 904b is connected to various components via an interface 905.

[0102] The input device 906 is, for example, a device that receives input from the user, such as a mouse, keyboard, touch panel, button, switch, or lever. The input device 906 may also be a microphone that detects the user's voice. The input device 906 may also be, for example, a remote control device that uses infrared or other radio waves, or an external connection device that corresponds to the operation of the information processing device 900.

[0103] The input device 906 further includes an input control circuit that outputs an input signal generated based on information input by the user to the CPU 901. By operating the input device 906, the user can input various types of data to the information processing device 900 or instruct it to perform processing operations.

[0104] The output device 907 is a device capable of presenting information acquired or generated by the information processing device 900 to the user visually or audibly. The output device 907 may be a display device such as an LCD (Liquid Crystal Display), PDP (Plasma Display Panel), OLED (Organic Light Emitting Diode) display, hologram, or projector; it may be an audio output device such as a speaker or headphones; or it may be a printing device such as a printer. The output device 907 can output the information obtained by processing of the information processing device 900 as text or images, or as sound.

[0105] The storage device 908 is a data storage device configured as an example of the storage unit of the information processing device 900. The storage device 908 may be composed of, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 can store programs executed by the CPU 901, various data, or various data acquired from external sources.

[0106] The drive 909 is a read or write device for removable recording media such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and is built into or external to the information processing device 900. For example, the drive 909 can read information recorded on the installed removable recording media and output it to the RAM 903. The drive 909 can also write data to the installed removable recording media.

[0107] The connection port 910 is a port for directly connecting an external device to the information processing device 900. The connection port 910 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. Alternatively, the connection port 910 may be an RS-232C port, an optical audio terminal, or an HDMI (High-Definition Multimedia Interface) port. By connecting the connection port 910 to an external device, various types of data can be transmitted and received between the information processing device 900 and the external device.

[0108] The communication device 911 is a communication interface composed of, for example, a communication device for connecting to a network N. The communication device 911 can send and receive signals using a predetermined protocol such as TCP / IP to, for example, the Internet or other communication devices. The communication device 911 may be, for example, a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or a WUSB (Wireless USB) communication card. Alternatively, the communication device 911 may be a router for optical communication, an ADSL (Asymmetric Digital Subscriber Line) router, or a modem for various types of communication.

[0109] Furthermore, it is possible to create programs that enable the CPU 901, ROM 902, RAM 903, and other hardware built into the computer to perform functions equivalent to those of the information processing device 900 described above. A computer-readable recording medium on which such a program is stored can also be provided.

[0110] While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art of the present disclosure that various modifications or alterations may be conceived within the scope of the technical ideas described in the claims, and these will naturally also fall within the technical scope of the present disclosure.

[0111] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0112] Furthermore, the following configurations also fall within the technical scope of this disclosure: (1) An information processing device comprising a processing unit that recognizes the start of an operation based on the overlap between a first detection target of a vehicle in an image obtained by a camera and a second detection target performing an operation on the vehicle. (2) The information processing device according to (1), wherein the first detection target is the opening of a door leading to the cargo bed of the vehicle. (3) The information processing device according to (2), wherein the operation is the loading and unloading of cargo from the cargo bed of the vehicle parked in a parking space. (4) The information processing device according to any one of (1) to (3), wherein the second detection target is the worker performing the operation or the work machine operated by the worker. (5) The information processing device according to any one of (1) to (4), wherein the processing unit recognizes the start of an operation based on at least one of the overlapping area of ​​the first detection target and the second detection target, or the elapsed time since the overlap occurred. (6) The information processing device according to any one of (1) to (5), wherein the overlap between the first detection target and the second detection target is the overlap between the bounding box of the first detection target and the bounding box of the second detection target. (7) The information processing device according to any one of (1) to (6), wherein the processing unit further recognizes the completion of the work based on the image obtained by the camera. (8) The information processing device according to (7), wherein the processing unit recognizes the completion of the work when the first detection target is no longer detected. (9) The information processing device according to (7) or (8), wherein the processing unit recognizes the completion of the work when it recognizes that the vehicle has left the parking space based on the size of the vehicle's body or the size of the vehicle's license plate in the image. (10) The information processing device according to any one of (7) to (9), further comprising an output unit that outputs the start time and end time of the recognized work.(11) An information processing system comprising: a processing unit that recognizes the start of work based on the overlap between a first detection target of a vehicle in an image obtained by a camera and a second detection target performing work on the vehicle; an output unit that outputs the recognized start time of the work; and a management unit that manages the outputted start time of the work. (12) The information processing system according to (11), wherein the processing unit further recognizes the end of the work based on the image obtained by the camera, and the management unit further manages the end time of the work further output from the output unit together with the start time. (13) The information processing system according to (12), wherein the management unit manages the start time and end time of the work in association with the vehicle number of the vehicle. (14) The information processing system according to (12) or (13), wherein the management unit further calculates the work time taken for the work based on the start time and end time of the work. (15) The information processing system according to (14), wherein the management unit further performs alert processing if the calculated work time is greater than or equal to a threshold. (16) The information processing system according to any one of (12) to (15), wherein the management unit further performs alert processing based on the difference between the start time and end time of the work and the reserved start time and end time of the work.

[0113] 1 Information Processing System 10 Camera 20 Management Server 110 Image Acquisition Unit 120 ISP Unit 130 Processing Unit 140 Output Unit 210 Communication Unit 220 Management Unit 230 Management Information Storage Unit N Network PA Parking Space TK Vehicle OD Opening WR Worker RD Rear Door TW Vehicle Body NP License Plate

Claims

An information processing device comprising a processing unit that recognizes the start of an operation based on the overlap between a first detection target of a vehicle in an image obtained by a camera and a second detection target on which an operation is performed on the vehicle. 　 The information processing device according to claim 1, wherein the first detection target is the opening of a door leading to the cargo bed of the vehicle. 　 The information processing device according to claim 2, wherein the operation is loading and unloading cargo from the cargo bed of the vehicle parked in the parking space. 　 The information processing device according to claim 1, wherein the second detection target is the worker performing the work or the work machine operated by the worker. 　 The information processing apparatus according to claim 1, wherein the processing unit recognizes the start of the operation based on at least one of the overlapping area between the first detection target and the second detection target, or the elapsed time since the overlap occurred. 　 The information processing apparatus according to claim 1, wherein the overlap between the first detection target and the second detection target is the overlap between the bounding box of the first detection target and the bounding box of the second detection target. 　 The information processing apparatus according to claim 1, wherein the processing unit further recognizes the completion of the operation based on the image obtained by the camera. 　 The information processing apparatus according to claim 7, wherein the processing unit recognizes the completion of the operation when the first detection target is no longer detected. 　 The information processing device according to claim 7, wherein the processing unit recognizes the completion of the operation when it recognizes that the vehicle has left the parking space based on the size of the vehicle's body or the size of the vehicle's license plate in the image. 　 The information processing apparatus according to claim 7, further comprising an output unit that outputs the start time and end time of the recognized operation. 　 A processing unit recognizes the start of the operation based on the overlap between a first detected target of the vehicle in the image obtained by the camera and a second detected target on which the operation is performed on the vehicle. An output unit that outputs the start time of the recognized operation, A management unit that manages the output start time of the aforementioned work, Information processing systems, including those mentioned above. 　 The processing unit further recognizes the completion of the operation based on the image obtained by the camera, The information processing system according to claim 11, wherein the management unit further manages the end time of the work output from the output unit together with the start time. 　 The information processing system according to claim 12, wherein the management unit manages the start time and end time of the work in association with the vehicle number of the vehicle. 　 The information processing system according to claim 12, wherein the management unit further calculates the time taken for the work based on the start time and end time of the work. 　 The information processing system according to claim 14, wherein the management unit further performs alert processing if the calculated work time is greater than or equal to a threshold. 　 The information processing system according to claim 12, wherein the management unit further performs alert processing based on the difference between the start time and end time of the work and the reserved start time and end time of the work.

Images