System

Abstract

A system is provided.SOLUTION: A system includes means for acquiring an image of surroundings, means for acquiring a rule of a parking lot, means for inputting a size and a specification of a vehicle, means for generating an optimal parking trajectory based on the acquired image and information on the vehicle, and means for automatically driving the vehicle or assisting a driver based on the generated parking trajectory.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For novice drivers and the elderly, parking safely and confidently in narrow parking spaces is a major challenge. Parking in a single space at the back of a parking lot is particularly challenging for inexperienced drivers, as it requires skill in determining the angle of entry, steering, and timing the backing up. Furthermore, the risk of collisions with surrounding obstacles and other vehicles increases when parking or exiting the parking lot. This invention is a system developed to solve these challenges. [Means for solving the problem]

[0005] The present invention is a system that includes the following means. First, it includes a means for acquiring images of the surroundings and a means for acquiring parking lot rules. Next, it includes a means for inputting the size and specifications of the vehicle and generating an optimal parking trajectory based on the acquired images and vehicle information. It also includes a means for automatically driving the vehicle based on the generated parking trajectory or a means for assisting the driver. It also includes a means for re-acquiring images of the surroundings when leaving the parking lot and generating an optimal trajectory for leaving. It also provides a means for displaying the generated parking trajectory and exit trajectory on an in-vehicle monitor to assist the driver. This enables even inexperienced drivers and elderly people to park safely and confidently in tight parking spaces.

[0006] "Surrounding images" refers to visual information about the parking lot and its surroundings, acquired using cameras and sensors installed at the entrance to the parking lot.

[0007] "Parking lot rules" are regulations and guidelines regarding the operation of parking lots, including no-entry areas, one-way rules, and information on available spaces.

[0008] "Vehicle size and specifications" refers to information about the physical characteristics of the vehicle to be parked, and includes elements such as length, width, height, and axle distance.

[0009] The "optimal parking trajectory" is the best route for a vehicle to safely and efficiently enter and park in a parking space, including the approach angle, steering operation, and timing of backing up.

[0010] "Means for autonomous driving" refers to the technology or system that enables a vehicle to automatically operate according to the generated parking trajectory and complete parking.

[0011] "Driver assistance means" refers to technologies and systems that display the generated parking trajectory on an in-vehicle monitor, provide visual and audio guidance to the driver, and assist in manual parking.

[0012] The "optimal exit trajectory" is the best route, including the direction of travel, steering, and timing of acceleration and deceleration, required for the vehicle to safely and efficiently exit the parking lot.

[0013] The "in-vehicle monitor" is a display device installed inside the vehicle, which visually displays the generated parking trajectory and exit trajectory to the driver. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention is a system that assists drivers in parking in tight spaces safely and efficiently. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automated driving or driver assistance.

[0036] Explanation of program processing

[0037] 1. Acquiring surrounding images and understanding parking lot rules

[0038] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[0039] 2. Enter your car information

[0040] The terminal provides a means for the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) so that the system can obtain accurate vehicle information to use in calculating the optimal parking trajectory.

[0041] 3. Parking Trajectory Generation

[0042] The server uses the captured images of the surroundings and vehicle information to generate the optimal parking trajectory, using an AI algorithm to calculate factors such as approach angle, steering operation, and timing for backing up, providing a route for safely and reliably entering a parking space.

[0043] 4. Implementation or assistance of automated driving

[0044] In Autonomous mode:

[0045] The device will then perform autonomous driving based on the generated optimal trajectory. In this mode, the device will automatically control steering, acceleration, and deceleration to complete the parking operation.

[0046] In driver assistance mode:

[0047] The device displays the generated optimal trajectory on an in-car monitor and assists the user in driving by following the displayed guidelines. The device also uses audio and visual alerts to support the user in taking appropriate actions at the appropriate time.

[0048] 5. New trajectory generation when leaving the warehouse

[0049] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates an optimal trajectory for leaving.

[0050] In Autonomous mode:

[0051] The terminal automatically drives the vehicle according to the generated departure trajectory and causes the vehicle to leave the parking lot.

[0052] In driver assistance mode:

[0053] The terminal displays the exit trajectory on the in-car monitor and assists the user in following the displayed guidelines. During exit, audio and visual alerts are also used to encourage appropriate operation.

[0054] Specific examples

[0055] Specific example of parking

[0056] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks the car in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly.

[0057] Specific example of shipping

[0058] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates a new optimal trajectory for leaving. The device then leaves the parking lot in autonomous driving mode or displays the trajectory in driver assistance mode, helping the user to follow it. This process allows the user to park and leave the parking lot with peace of mind.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] The server acquires current surrounding images from a camera installed at the entrance of the parking lot, which are used to recognize the layout of parking spaces and the location of obstacles.

[0062] Step 2:

[0063] The server collects information on parking rules and available spaces from the database, which allows users to know the exact location of available spaces and important points to note when parking.

[0064] Step 3:

[0065] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[0066] Step 4:

[0067] The device sends the entered vehicle information to the server, which then receives all the data necessary for analysis.

[0068] Step 5:

[0069] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[0070] Step 6:

[0071] The server then transmits the generated optimal parking trajectory data to the terminal, allowing the terminal to learn the specific parking method.

[0072] Step 7:

[0073] When parking in autonomous driving mode based on the received parking trajectory, the device automatically performs steering, acceleration, and deceleration, thereby completing parking safely and efficiently.

[0074] Step 8:

[0075] When the device is in driver assistance mode, the generated parking trajectory is displayed on the in-car monitor. The user drives and parks according to this display. The device prompts the user to operate the vehicle at the appropriate time using audio and visual alerts.

[0076] Step 9:

[0077] After parking is complete, the terminal notifies the user, allowing the user to confirm that parking was completed successfully.

[0078] Step 10:

[0079] To start preparation for delivery, the user issues an instruction to the system, which makes the server ready to acquire new information.

[0080] Step 11:

[0081] The server then acquires images of the surrounding area again to obtain the latest information, thereby ensuring safety when the vehicle leaves the warehouse.

[0082] Step 12:

[0083] The server generates an optimal trajectory for the departure, calculating a new departure route and taking into account the location of obstacles and the movement of other vehicles.

[0084] Step 13:

[0085] The server then transmits the generated retrieval trajectory data to the terminal, which then retrieves the vehicle based on this information.

[0086] Step 14:

[0087] The terminal either drives automatically along the departure trajectory or displays the information on a monitor to assist the user in leaving the parking lot. In the autonomous driving mode, the terminal performs all operations automatically. In the driver assistance mode, the user operates the device themselves.

[0088] Step 15:

[0089] After the vehicle has left the parking lot, the terminal notifies the user, allowing the user to confirm that the vehicle has safely left the parking lot.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] Current parking assistance systems lack the precision and convenience required to achieve safe and efficient parking in tight parking spaces. The ability to quickly respond to changes in the surrounding environment is particularly important, and safety when exiting the parking space is equally important. Furthermore, the provision of interfaces and guidance to reduce the driver's burden is insufficient. The present invention aims to solve these problems.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes a means for acquiring surrounding conditions, a means for acquiring parking lot rule information, a means for inputting detailed vehicle information, a means for generating an optimal parking route based on the acquired images and vehicle information, and a means for automatically operating the vehicle or assisting the driver based on the generated parking route, thereby enabling the user to safely and efficiently park and exit even in a narrow parking space.

[0095] "Surrounding conditions" refers to information that indicates the state of objects and space around the vehicle.

[0096] "Parking lot rule information" refers to data regarding parking rules and available spaces that apply within a parking lot.

[0097] "Vehicle details" refers to specific data about the vehicle's dimensions and specifications (e.g., length, width, height, distance between axles, etc.).

[0098] An "optimal parking route" is the best calculated path for a vehicle to safely and efficiently enter a parking space.

[0099] "Autonomous operation" means that the system automatically controls the operation of the vehicle, including steering, acceleration, and deceleration.

[0100] "Means to assist the driver" refers to the system's ability to provide appropriate driving assistance to the driver in the form of voice or visual alerts, guidelines, etc.

[0101] An "optimal route for exit" is the calculated best path for a vehicle to safely exit a parking space.

[0102] An "in-vehicle display device" refers to a monitor or display device installed inside a vehicle that displays information to the driver.

[0103] The present invention is a system that assists drivers in parking and leaving narrow parking spaces safely and efficiently. The system includes a series of means for acquiring the surrounding situation, understanding parking lot rule information, inputting detailed vehicle information, generating optimal parking and leaving routes, and automatically operating the vehicle or assisting the driver based on the generated routes.

[0104] First, the server acquires the surrounding situation using a high-resolution camera installed at the entrance to the parking lot. This camera captures the surrounding image in real time and recognizes the layout of parking spaces and the location of obstacles using an object detection algorithm based on deep learning. The server also obtains information on parking lot rules and available spaces from a database. This enables accurate calculation of the parking route.

[0105] Next, the device provides a means for the user to input detailed vehicle information (e.g., length, width, height, axle distance, etc.). The input information is stored in a database and made available to the server. Based on this information, the server generates an optimal parking route. Specifically, it uses AI algorithms to calculate entry angles, steering operations, timing of backing, etc., and provides a route for safely and reliably entering a parking space. Specific technologies used include deep learning and reinforcement learning.

[0106] Based on the generated parking route, the device parks in either autonomous mode or driver-assisted mode. In autonomous mode, the device automatically controls steering, acceleration, and deceleration to complete the parking. In driver-assisted mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. Audio and visual alerts are used to help the user perform operations at the appropriate time.

[0107] After parking, when the user begins preparing to leave, the server again acquires information about the surrounding area and generates the optimal route for leaving. Both autonomous driving mode and driver assistance mode are available during leaving, allowing the user to leave the parking lot with peace of mind.

[0108] For example, you can instruct a generative AI model using specific prompts like the following:

[0109] 1. "Describe an automated driving assistance system for tight parking spaces."

[0110] 2. "How can I calculate the optimal parking route using vehicle and parking information?"

[0111] 3. "Please explain the difference between autonomous and driver assistance modes."

[0112] These prompts allow the system to properly obtain the necessary information and provide optimal parking and exit assistance.

[0113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0114] Step 1: Acquire an image of the surroundings

[0115] The server acquires images of the surrounding area using a high-resolution camera installed at the entrance to the parking lot. This camera captures images in real time, and the server receives the image data. The acquired image data is analyzed using a deep learning algorithm to recognize the layout of parking spaces and the location of obstacles. The input is the camera's image data, and the output is the analyzed image data.

[0116] Step 2: Know the parking rules

[0117] The server retrieves rules information from a parking lot database. This database contains parking rules and available space information. For example, it checks whether a particular parking space is reserved or whether an obstacle exists. The input is a request for parking rules information, and the output is the parking rules and available space information.

[0118] Step 3: Enter vehicle information

[0119] The terminal provides a means for the user to input vehicle details (length, width, height, axle distance, etc.). The user enters these details through an input form. The input is the vehicle information entered by the user, and the output is the vehicle information stored in a database.

[0120] Step 4: Generate optimal parking trajectory

[0121] The server generates the optimal parking trajectory based on the surrounding image obtained in step 1, the parking lot rules acquired in step 2, and the detailed vehicle information entered in step 3. It uses AI algorithms (including deep learning and reinforcement learning) to calculate the approach angle, steering operation, timing of backing up, etc. The inputs are image data, parking lot rules, and vehicle information, and the output is the optimal parking route.

[0122] Step 5: Implementing or Assisting Autonomous Driving

[0123] In Autonomous mode:

[0124] The terminal performs autonomous driving based on the optimal parking trajectory received from the server. It controls the engine, steering, brakes, etc., and automatically parks the vehicle according to the instructed trajectory. The input is the optimal parking route, and the output is the completion of automatic parking.

[0125] In driver assistance mode:

[0126] The terminal displays the generated optimal trajectory on the monitor inside the vehicle and assists the user in parking by following the displayed guidelines. Guidelines for approach angles and steering operations are displayed on the navigation screen, and audio and visual alerts are used to prompt the user to perform operations at the appropriate time. The input is the optimal parking route, and the output is the completion of driver assistance.

[0127] Step 6: New trajectory generation for exit

[0128] When the user starts preparing to leave, the server acquires images of the surroundings again and generates an optimal trajectory for leaving. It calculates a safe route to leave, taking into account the position information of other vehicles and obstacles in the parking lot. The input is the latest surrounding images and information about the parking lot, and the output is the optimal route to leave.

[0129] Step 7: Conduct or assist with shipping

[0130] In Autonomous mode:

[0131] The terminal automatically drives the vehicle based on the generated departure trajectory and causes it to leave the parking lot. The input is the optimal departure route, and the output is the completion of automatic departure.

[0132] In driver assistance mode:

[0133] The terminal displays the departure trajectory on the in-vehicle monitor and assists the user in following the displayed guidelines. Audio and visual alerts are used to prompt appropriate operation. The input is the optimal departure route, and the output is completion of the driver's assistance with leaving the vehicle.

[0134] (Application example 1)

[0135] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] It is often difficult for drivers to safely and efficiently park in narrow parking spaces, especially in urban areas and crowded parking lots. Current parking assistance systems are insufficient, and further support is needed to enable drivers to park and exit safely. Another issue is the lack of technology that can identify available parking spaces and the location of obstacles in a parking lot in real time and provide the optimal parking method.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0138] In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting the size and specifications of the vehicle, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for executing an artificial intelligence algorithm for recognizing the acquired images of the surroundings of the parking lot, and means for generating parking guidelines based on the recognized position information of vacant spaces and obstacles in the parking lot and displaying them on a screen. This enables the driver to park and exit a narrow parking space safely and efficiently with peace of mind.

[0139] The "means for acquiring images of the surroundings" refers to a means for collecting image data using a camera or a sensor in order to grasp the surrounding environment of the vehicle.

[0140] "Means for obtaining parking lot rules" refers to a means for obtaining information such as parking lot usage rules, layout, and available spaces from a database.

[0141] "Means for inputting vehicle size and specifications" refers to a means for a user to input detailed information such as the length, width, height, and axle distance of a vehicle.

[0142] The "means for generating the optimal parking trajectory" is a means for calculating and generating a safe and efficient parking route using an AI algorithm based on acquired image data and vehicle information.

[0143] "Means for automatically driving a vehicle or assisting a driver" refers to means including a function for automatically controlling the operation of a vehicle based on the generated parking trajectory, or a function for providing guidelines or voice guidance to support the driver when driving by himself.

[0144] "Means for executing artificial intelligence algorithms" refers to means for using AI technology to analyze images of the surrounding area of ​​the parking lot and identify the location of vacant spaces and obstacles.

[0145] "Means for generating parking guidelines and displaying them on the screen" refers to a means for generating guidelines for the driver to follow based on the recognized parking lot information and displaying them on a smartphone or in-car display.

[0146] The present invention is a system for assisting drivers in parking and leaving narrow parking spaces safely and efficiently. The system includes the following main means:

[0147] 1. A means of capturing images of the surroundings

[0148] The server collects image data using cameras and sensors to understand the surrounding environment of the parking lot. These cameras are installed at the entrance to the parking lot and around each parking space. The image data is acquired in real time and sent to the system.

[0149] 2. How to obtain parking rules

[0150] The server retrieves information about the parking lot, such as usage rules, layout, available spaces, etc. from a database. This information is regularly updated by the parking lot manager and provided to the server.

[0151] 3. A means of inputting vehicle size and specifications

[0152] Users can input detailed information such as vehicle length, width, height, and axle distance through the terminal, which is used to calculate the optimal parking trajectory.

[0153] 4. Means for generating optimal parking trajectories

[0154] The server uses an AI algorithm to calculate and generate a safe and efficient parking route based on the acquired image data and vehicle information. This algorithm determines the optimal trajectory by taking into account factors such as approach angle, steering operation, and timing of backing up.

[0155] 5. Means for autonomous driving or driver assistance of vehicles

[0156] The device provides a function to automatically control vehicle operation based on the generated parking trajectory. It also supports the driver by providing parking guidelines and voice guidance when driving independently. This function displays parking guidelines on a smartphone or in-car display to help the driver park according to them.

[0157] 6. Means for running artificial intelligence algorithms

[0158] The server uses AI technology to analyze images of the surrounding parking lot and identify the location of available spaces and obstacles. This AI algorithm uses a deep learning model to suggest the optimal parking strategy based on the results of the image analysis.

[0159] 7. A method for generating parking guidelines and displaying them on the screen

[0160] The device generates guidelines for the driver to follow based on the recognized parking information, displays them on a smartphone or in-car display, and uses voice synthesis to provide driving instructions at the appropriate time.

[0161] Specific examples

[0162] For example, a user arrives at a small parking lot in central Tokyo and launches a parking support app on their smartphone. The app scans the entire parking lot using a camera and calculates the optimal parking route using an AI algorithm. The user parks their car according to the guidelines displayed on the smartphone screen and uses voice guidance to perform the appropriate operations, completing the parking process with peace of mind. An example of a prompt that can be applied at this time is, "Please recognize the parking lot layout and identify the location of available spaces and obstacles."

[0163] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0164] Step 1:

[0165] Input: Real-time image data from a parking lot camera.

[0166] Processing: The server uses the parking lot cameras to capture images of the surrounding area, including footage of the parking space and any obstacles in the area.

[0167] Output: Captured parking lot image data.

[0168] Step 2:

[0169] Input: Parking rules, available space information, and image data.

[0170] Processing: The server collects information about parking rules and available spaces from a database and compares it with the captured image data, for example, identifying no-parking areas and the location of obstacles.

[0171] Output: Parking lot layout information as a result of image processing (available parking areas, empty spaces, obstacle locations).

[0172] Step 3:

[0173] Input: Vehicle size and specifications (length, width, height, axle distance).

[0174] Processing: The user uses a device (smartphone) to input vehicle information, which is then sent to the server.

[0175] Output: Transmitted vehicle information data.

[0176] Step 4:

[0177] Input: Parking lot layout information, vehicle information.

[0178] Processing: The server uses AI algorithms to generate the optimal parking trajectory based on the input vehicle information and parking lot layout information, for example, calculating the parking space entry angle and backing timing.

[0179] Output: Generated parking trajectory data.

[0180] Step 5:

[0181] Input: Parking trajectory data.

[0182] Processing: Based on the generated optimal parking trajectory, the device generates parking guidelines to assist the driver and displays them on a smartphone or in-car display. It also uses the voice guidance function to provide driving instructions at the appropriate time.

[0183] Output: Parking guideline display data and voice guidance data provided to the user.

[0184] Step 6:

[0185] Input: User action to prepare for delivery.

[0186] Processing: When the user begins preparing to leave, the server again captures images of the surroundings and uses AI algorithms to generate an optimal exit trajectory, taking into account any new obstacles or changing parking lot conditions.

[0187] Output: Generated outbound trajectory data.

[0188] Step 7:

[0189] Input: Outbound trajectory data.

[0190] Processing: Based on the generated departure trajectory, the terminal generates departure guidelines to assist the driver and displays them on a smartphone or in-car display. It also uses a voice guidance function to provide departure instructions at the appropriate time.

[0191] Output: Display data and voice guidance data for delivery guidelines provided to the user.

[0192] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0193] The present invention is a system that assists drivers in parking in tight spaces safely and efficiently, and further combines it with an emotion engine that recognizes the user's emotions. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automatic driving or driver assistance, as well as analyzing the user's emotions.

[0194] Explanation of program processing

[0195] 1. Acquiring surrounding images and understanding parking lot rules

[0196] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[0197] 2. Enter your car information

[0198] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[0199] 3. Parking Trajectory Generation

[0200] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[0201] 4. Implementation or assistance of automated driving

[0202] In Autonomous mode:

[0203] The device will then perform autonomous driving based on the generated optimal trajectory. In this mode, the device will automatically control steering, acceleration, and deceleration to complete the parking operation.

[0204] In driver assistance mode:

[0205] The device displays the generated optimal trajectory on an in-car monitor and assists the user in driving by following the displayed guidelines. The device also prompts the user to take action at the appropriate time using audio and visual alerts.

[0206] 5. User Emotion Recognition

[0207] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling stressed or anxious, that emotion is conveyed to the system.

[0208] 6. Emotion-based system adjustment

[0209] The server adjusts the system's behavior based on the recognized user's emotions.

[0210] If the user is feeling stressed, the device will play relaxing music and gently guide them through operations.

[0211] If the user feels anxious or nervous, the device will preferentially select the driving assistance mode and provide detailed guidance on the parking trajectory.

[0212] 7. New trajectory generation when leaving the warehouse

[0213] After parking, when the user starts preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. Based on the input information, it generates the optimal trajectory for leaving the parking lot.

[0214] 8. Conducting or assisting with deliveries

[0215] In Autonomous mode:

[0216] The terminal automatically drives the vehicle according to the generated departure trajectory and causes the vehicle to leave the parking lot.

[0217] In driver assistance mode:

[0218] The terminal displays the exit trajectory on the in-car monitor and guides the user through the exit process by following the displayed guidelines, using audio and visual alerts to encourage appropriate operation.

[0219] Specific examples

[0220] Specific example of parking

[0221] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly. The device also recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[0222] Specific example of shipping

[0223] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates a new optimal trajectory for leaving. The device then either leaves the vehicle in autonomous driving mode or displays the trajectory in driver assistance mode and helps the user leave the vehicle accordingly. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[0224] The processing flow will be explained below.

[0225] Step 1:

[0226] The server acquires current surrounding images from a camera installed at the entrance of the parking lot, which are used to recognize the layout of parking spaces and the location of obstacles.

[0227] Step 2:

[0228] The server collects information on parking rules and available spaces from the database, which allows users to know the exact location of available spaces and important points to note when parking.

[0229] Step 3:

[0230] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[0231] Step 4:

[0232] The device sends the entered vehicle information to the server, which then receives all the data necessary for analysis.

[0233] Step 5:

[0234] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[0235] Step 6:

[0236] The server then transmits the generated optimal parking trajectory data to the terminal, allowing the terminal to learn the specific parking method.

[0237] Step 7:

[0238] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, it uses facial expression analysis and voice analysis technology to detect whether the user is feeling stressed or anxious.

[0239] Step 8:

[0240] When the emotion engine recognizes emotions such as stress or anxiety, the device automatically adjusts system behavior, including playing relaxing music and providing more detailed operating instructions.

[0241] Step 9:

[0242] When parking in autonomous driving mode based on the received parking trajectory, the device automatically performs steering, acceleration, and deceleration, thereby completing parking safely and efficiently.

[0243] Step 10:

[0244] When the device is in driver assistance mode, the generated parking trajectory is displayed on the in-car monitor. The user drives and parks according to this display. The device prompts the user to operate the vehicle at the appropriate time using audio and visual alerts.

[0245] Step 11:

[0246] After parking is complete, the terminal notifies the user, allowing the user to confirm that parking was completed successfully.

[0247] Step 12:

[0248] To start preparation for delivery, the user issues an instruction to the system, which makes the server ready to acquire new information.

[0249] Step 13:

[0250] The server then acquires images of the surrounding area again to obtain the latest information, thereby ensuring safety when the vehicle leaves the warehouse.

[0251] Step 14:

[0252] The server generates an optimal trajectory for the departure, calculating a new departure route and taking into account the location of obstacles and the movement of other vehicles.

[0253] Step 15:

[0254] The server then transmits the generated retrieval trajectory data to the terminal, which then retrieves the vehicle based on this information.

[0255] Step 16:

[0256] The terminal either drives automatically along the departure trajectory or displays the information on a monitor to assist the user in leaving the parking lot. In the autonomous driving mode, the terminal performs all operations automatically. In the driver assistance mode, the user operates the device themselves.

[0257] Step 17:

[0258] After the vehicle has left the parking lot, the terminal notifies the user, allowing the user to confirm that the vehicle has safely left the parking lot.

[0259] Example 2

[0260] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0261] Parking safely and efficiently in tight parking spaces is a challenging task for many drivers. The stress and anxiety drivers feel during parking also impact safety. However, current automated driving and driver assistance systems lack the ability to recognize user emotions and adapt to the user's psychological state during the parking process. Therefore, there is a need for a parking assistance system that can adjust its behavior based on the user's emotions.

[0262] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0263] In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting the size and specifications of the vehicle, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for recognizing the user's emotions, and means for adjusting the system operation based on the recognized user's emotions, thereby making it possible to provide optimal parking support while taking the user's emotions into consideration.

[0264] "Means for acquiring images of the surrounding area" refers to technology that collects real-time images of the surrounding area using cameras and sensors installed at the entrance to the parking lot.

[0265] "Means for obtaining parking lot rules" refers to technology for obtaining traffic rules and available space information within a parking lot from a database.

[0266] "Means for inputting vehicle dimensions and specifications" refers to technology that provides an interface for users to input the dimensions and specifications (e.g., length, width, height, axle distance, etc.) of their vehicle.

[0267] "Means for generating optimal parking trajectories" refers to technology that uses AI algorithms to calculate entry angles, steering operation, timing of backing up, etc. based on acquired surrounding images and vehicle information to generate optimal parking trajectories.

[0268] "Means for autonomously driving a vehicle or assisting the driver" refers to technology that automatically controls the vehicle's steering, acceleration, and deceleration based on the generated parking trajectory, or technology that displays the trajectory on an in-vehicle monitor and assists the driver with audio and visual alerts.

[0269] "Means for recognizing user emotions" refers to an emotion engine that analyzes emotions such as stress and anxiety from the user's facial expressions and voice.

[0270] "Means for adjusting system behavior" refers to functions that change the behavior of the system based on the recognized user's emotions, such as playing relaxing music or providing gentler operating guidance.

[0271] This invention combines a system that assists drivers in parking in tight spaces safely and efficiently with an emotion engine that recognizes the user's emotions. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automatic driving or driver assistance, as well as analyzing the user's emotions.

[0272] The detailed system configuration uses the following hardware and software:

[0273] Acquiring surrounding images and understanding parking rules

[0274] The server obtains current surrounding images from a camera installed at the entrance to the parking lot. This high-resolution camera can clearly recognize obstacles and available spaces within the parking lot. Image analysis is performed in real time using image recognition software such as OpenCV. At the same time, the server collects information on parking lot rules and available spaces from a database. This database records traffic rules within the parking lot and space usage status.

[0275] Enter your car information

[0276] The device provides an interface for the user to input vehicle dimensions and specifications (e.g., length, width, height, and axle distance). This interface may take the form of a smartphone app or an in-vehicle display. The input information is stored on the device and sent to a server to avoid having to enter it again.

[0277] Parking trajectory generation

[0278] The server performs data analysis based on the acquired surrounding images and vehicle information. AI algorithms (e.g., TensorFlow and PyTorch) are used for the analysis, calculating factors such as approach angle, steering operation, and timing of backing up, and generating the optimal parking trajectory. The generated trajectory information is sent to the device.

[0279] Implementation or assistance of automated driving

[0280] In autonomous driving mode, the device automatically drives the vehicle based on the generated optimal trajectory. In this mode, the device automatically controls steering, acceleration, and deceleration, and completes parking using dedicated autonomous driving software (e.g., Autoware).

[0281] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. The device uses audio and visual alerts to prompt the user to take action at the appropriate time.

[0282] User Emotion Recognition

[0283] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. Emotion analysis utilizes facial and voice analysis software such as Microsoft Azure Face API and Google Speech-to-Text. For example, if the user is feeling stressed or anxious, that emotion is conveyed to the system.

[0284] Emotion-based system adjustment

[0285] The server adjusts the system's behavior based on the user's recognized emotions. Specifically, if the user feels stressed, the device will play relaxing music (e.g., using Spotify or Apple Music APIs) and provide gentle guidance on operations. If the user feels anxious or tense, the device will prioritize driving assistance mode and provide detailed guidelines.

[0286] New trajectory generation when leaving the warehouse

[0287] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area to ascertain the latest situation. Based on the input information, an AI algorithm is used to generate the optimal trajectory for leaving the parking lot and send it to the terminal.

[0288] Conducting or assisting with shipping

[0289] In autonomous driving mode, the device automatically drives the vehicle according to the generated exit trajectory and leaves the parking lot. In driver assistance mode, the device displays the exit trajectory on the in-car monitor and helps the user leave the parking lot by following the displayed guidelines. Audio and visual alerts are also used during this process.

[0290] Specific examples

[0291] As a concrete example of parking, when a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device that receives this information either parks in autonomous driving mode or displays the trajectory on a monitor in driver assistance mode, helping the user park accordingly. At the same time, the device recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[0292] Example prompts to input to the generative AI model:

[0293] "A user arrives at a parking lot, and the system automatically finds a parking space and assists the user. If the user is emotionally stressed, how will the system respond?"

[0294] For example, when the user prepares to leave the parking lot after parking, the server acquires images of the surrounding area again and generates a new optimal trajectory for leaving. The device then either leaves the parking lot in autonomous driving mode or displays the trajectory in driver assistance mode and assists the user in leaving the parking lot. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[0295] Example prompts to input to the generative AI model:

[0296] "When the user attempts to leave the parking lot after parking, please explain how the system will assist the user in leaving the parking lot and how it will respond depending on the user's emotions."

[0297] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0298] Step 1: Acquire surrounding images and understand parking rules

[0299] The server obtains real-time images of the surrounding area from a camera installed at the entrance to the parking lot. The camera image is used as input, and the obtained image data is obtained as output. This image data is analyzed to recognize the layout of parking spaces and the location of obstacles. Image recognition software such as OpenCV is used for image analysis. At the same time, the server obtains information on parking lot rules and available spaces from a database. The information registered by the user at the parking lot is used as input, and the obtained parking lot rule data is obtained as output.

[0300] Specific behavior:

[0301] The server receives the video stream from the camera.

[0302] The server performs image analysis and recognizes parking spaces and obstacles.

[0303] The server retrieves the parking rules from the database.

[0304] Step 2: Enter your vehicle information

[0305] The terminal allows the user to input the vehicle size and specifications (length, width, height, axle distance, etc.). The user's input data is used as input, and stored vehicle information is obtained as output. This information is saved on the terminal and later sent to the server. The stored data is reused to avoid having to enter the information again.

[0306] Specific behavior:

[0307] The device displays the vehicle information input screen.

[0308] The user inputs the vehicle size and specifications.

[0309] The device saves the input data and sends it to the server.

[0310] Step 3: Generate parking trajectories

[0311] The server performs data analysis based on the acquired surrounding images and the user's vehicle information. Surrounding image data and vehicle information data are used as input, and parking trajectory data is generated as output. Here, AI algorithms (e.g., TensorFlow and PyTorch) are used to calculate factors such as approach angle, steering operation, and timing of backing up, and generate the optimal parking trajectory.

[0312] Specific behavior:

[0313] The server acquires surrounding images and vehicle information.

[0314] The server analyzes using an AI algorithm.

[0315] The server generates the optimal parking trajectory and sends it to the device.

[0316] Step 4: Implementing or Assisting Autonomous Driving

[0317] In autonomous driving mode, the device performs autonomous driving based on the generated optimal trajectory. Parking trajectory data is used as input, and the parking completion status is obtained as output. Dedicated autonomous driving software (e.g., Autoware) is used.

[0318] Specific behavior:

[0319] The terminal receives the optimal orbit data.

[0320] The device begins autonomous driving using Autoware.

[0321] The device controls the steering wheel, accelerator, and brakes.

[0322] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user to drive according to the guideline. The input is the parking trajectory data, and the output is driving assistance for the user. The device uses audio and visual alerts.

[0323] Specific behavior:

[0324] The terminal displays the optimal orbit data on the monitor.

[0325] The user drives according to the guidelines.

[0326] The device assists the user with voice alerts.

[0327] Step 5: Recognizing User Emotions

[0328] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. The user's facial expression and voice data are used as input, and the recognized emotion data is obtained as output. Emotion analysis uses Microsoft Azure Face API and Google Speech-to-Text, among others.

[0329] Specific behavior:

[0330] The device uses a camera and microphone to collect the user's facial expressions and voice.

[0331] The device analyzes using an emotion engine.

[0332] The device sends the analysis results to the server.

[0333] Step 6: Adjusting the system based on emotions

[0334] The server adjusts the system's behavior based on the recognized user emotion. Emotional data is used as input and the adjusted system behavior is obtained as output. If the user is feeling stressed, the device will play relaxing music (using Spotify or Apple Music APIs) or provide gentle guidance.

[0335] Specific behavior:

[0336] The server processes the emotion data.

[0337] The device will select and play the appropriate music.

[0338] The device provides a friendly audio guide.

[0339] Step 7: New trajectory generation for exit

[0340] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. The surrounding image data is used as input, and the optimal trajectory data for leaving is obtained as output. The optimal trajectory for leaving is generated using an AI algorithm and sent to the terminal.

[0341] Specific behavior:

[0342] The user begins preparation for delivery.

[0343] The server acquires surrounding images from the camera.

[0344] The server generates the optimal trajectory for delivery and sends it to the terminal.

[0345] Step 8: Perform or assist with shipping

[0346] In the automatic driving mode, the terminal automatically drives according to the generated retrieval trajectory. The retrieval trajectory data is used as input, and the retrieval completion status is obtained as output.

[0347] Specific behavior:

[0348] The terminal receives the delivery trajectory data.

[0349] The terminal will execute the retrieval in automatic operation mode.

[0350] In driver assistance mode, the device displays the exit trajectory on the in-vehicle monitor and assists the user in following the guidelines when leaving the vehicle. The exit trajectory data is used as input, and driving assistance is provided to the user as output. Audio and visual alerts encourage appropriate operation.

[0351] Specific behavior:

[0352] The terminal displays the delivery trajectory data on the monitor.

[0353] The user issues goods according to the guidelines.

[0354] The device will prompt you with an audio alert.

[0355] (Application example 2)

[0356] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] Modern parking lots often have narrow spaces and complex layouts, making parking difficult for drivers. The stress and anxiety drivers feel when parking can negatively impact driving safety and efficiency. Even in these situations, there is a demand for systems that support safe and efficient parking. Furthermore, systems must provide support not only when parking but also when exiting the parking lot safely and efficiently.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting vehicle size and specifications, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for analyzing user emotions, and means for adjusting system operation based on the user emotions. This enables the driver to park safely and efficiently, and by providing support according to the driver's emotions, it becomes possible to reduce stress and anxiety.

[0359] The "surrounding image" is an image acquired using a device such as a camera to recognize the layout of the parking lot and the location of obstacles.

[0360] "Parking rules" are rules and instructions regarding parking spaces, prohibited areas, and how to pass through a parking lot.

[0361] "Vehicle size and specifications" refers to information that refers to the physical dimensions and characteristics of a vehicle, such as length, width, height, and axle distance.

[0362] A "parking trajectory" is a planned path for a vehicle to park in a parking space, including the optimal approach angle, steering operation, and timing of backing up.

[0363] "Autonomous driving" is the ability of a vehicle to perform driving operations autonomously without human intervention.

[0364] "Supporting the driver" means providing assistance by providing guidelines, alerts, etc. when the driver parks or leaves the vehicle by themselves.

[0365] "User's emotions" refer to psychological states such as stress, anxiety, joy, etc. that are recognized from the user's facial expressions, voice, etc.

[0366] "Analyzing emotions" refers to the process of analyzing the user's facial expressions and voice to identify their emotions.

[0367] "Adjusting the system's behavior" means changing the system's functions and behavior in response to the user's emotions, for example, by playing relaxing music or providing gentle guidance.

[0368] This invention relates to a system that assists drivers in parking in narrow parking spaces safely and efficiently. The system includes multiple means for capturing images of the surrounding area, understanding parking rules, inputting vehicle information, generating an optimal parking trajectory, and performing automated driving or driver assistance and user emotion analysis.

[0369] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[0370] The terminal prompts the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.). This information is stored in the terminal, eliminating the need to input it again. The server generates the optimal parking trajectory based on the captured surrounding images and vehicle information. The server uses an AI algorithm (e.g., TensorFlow) to calculate factors such as approach angle, steering operation, and timing for backing up, and generates the optimal parking trajectory.

[0371] In autonomous driving mode, the device drives autonomously based on the generated optimal trajectory. In this mode, the device automatically controls steering, acceleration, and deceleration to complete parking. In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. The device uses audio and visual alerts to prompt the user to operate at the appropriate time.

[0372] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling stressed or anxious, that emotion is communicated to the system. The server adjusts the system's operation based on the recognized user emotion. If the user is feeling stressed, the device may play relaxing music or gently guide the user through the operation. If the user is feeling anxious or tense, the device may prioritize driving assistance mode and provide detailed guidance on the parking trajectory.

[0373] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area and ascertains the latest situation. Based on the input information, it generates an optimal trajectory for leaving. In autonomous driving mode, the device automatically drives the vehicle according to the generated trajectory and leaves the parking lot. In driver assistance mode, the device displays the trajectory on the in-car monitor and assists the user in leaving the parking lot by following the displayed guidelines. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[0374] Specific examples

[0375] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly. The device also recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[0376] Prompt Sentence Examples

[0377] "A user is trying to park a car in a tight parking space. Implement a system that recognizes whether the user is stressed or not from their facial expression, and generates an optimal parking trajectory to assist the user in parking in real time."

[0378] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0379] Step 1:

[0380] The server obtains current surrounding images from a camera installed at the entrance to the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. The video data obtained from the camera (input) is used to identify the location of vacant spaces in the parking lot and obstacles (example of data processing: object detection using image processing algorithms). The output becomes map information of the parking lot.

[0381] Step 2:

[0382] The server collects information about parking rules and available spaces from a database. This information (input) includes rules and instructions about permitted parking spaces, prohibited areas, and how to navigate. The output is the optimal parking location that complies with the permitted parking areas and rules. This is achieved by executing a database query to obtain the required information.

[0383] Step 3:

[0384] The terminal allows the user to input the vehicle size and specifications (e.g. length, width, height, axle distance, etc.). This information (input) is stored in a vehicle database. The (output) is the complete vehicle dimensions, held in a reusable format. Input is received through a user interface.

[0385] Step 4:

[0386] The server generates an optimal parking trajectory based on the acquired surrounding images and vehicle information. It uses an AI algorithm (e.g., TensorFlow) to calculate the approach angle, steering operation, timing of backing up, etc. (example of data calculation: application of a trajectory prediction model using a neural network) and generates the generated parking trajectory (output).

[0387] Step 5:

[0388] In autonomous driving mode, the device performs autonomous driving based on the generated optimal trajectory. It automatically controls steering, acceleration, and deceleration (vehicle operation using a control algorithm) and completes parking (output). The input is optimal trajectory information. Sensor information is fed back in real time to control the vehicle's movement.

[0389] Step 6:

[0390] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. It uses audio and visual alerts to prompt the user to operate the vehicle at the appropriate time (input is the generated parking trajectory information, output is guideline display and alerts) to complete parking. It generates audio and visual guides and displays them on the user interface.

[0391] Step 7:

[0392] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. It analyzes the user's facial images and voice data (input) (example of data processing: facial expression and voice analysis using machine learning models) and identifies the user's state of stress or anxiety (output). It then identifies the user's emotional state based on the results of the data analysis.

[0393] Step 8:

[0394] The server adjusts the system's behavior based on the recognized user emotion. If the user feels stressed, the device plays relaxing music (input: emotion recognition result, output: music playback) and gently guides the user through the operation. If the user feels anxious or tense, the device prioritizes driving assistance mode and provides more detailed guidance (output), adjusts system settings, and provides appropriate feedback.

[0395] Step 9:

[0396] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. The acquired image data (input) is used to re-recognize the surrounding environment (example of data processing: re-application of object detection algorithm) and generate the optimal trajectory for leaving (output). The optimal exit route is calculated through data analysis.

[0397] Step 10:

[0398] In autonomous driving mode, the terminal automatically drives the vehicle according to the generated departure trajectory and leaves the parking lot. It performs autonomous driving control based on the departure trajectory information (input) and optimizes the vehicle's movement (output). It executes departure operations including real-time feedback control.

[0399] Step 11:

[0400] In driver assistance mode, the device displays the exit trajectory on the in-car monitor and assists the user in following the displayed guidelines to complete the exit. Audio and visual alerts are used to prompt the user to take the appropriate action and complete the exit (input is the exit trajectory information, output is the guideline display and alerts). The device makes full use of the user interface to provide appropriate guidance and feedback.

[0401] Example prompt: "A user is trying to park a car in a narrow parking space. Implement a system that recognizes whether the user is stressed from their facial expression and generates an optimal parking trajectory to assist the user in parking in real time."

[0402] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0403] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0404] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0405] [Second embodiment]

[0406] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0407] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0408] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0409] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0410] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0412] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0413] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0414] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0415] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0416] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0417] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0418] The present invention is a system that assists drivers in parking in tight spaces safely and efficiently. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automated driving or driver assistance.

[0419] Explanation of program processing

[0420] 1. Acquiring surrounding images and understanding parking lot rules

[0421] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[0422] 2. Enter your car information

[0423] The terminal provides a means for the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) so that the system can obtain accurate vehicle information to use in calculating the optimal parking trajectory.

[0424] 3. Parking Trajectory Generation

[0425] The server uses the captured images of the surroundings and vehicle information to generate the optimal parking trajectory, using an AI algorithm to calculate factors such as approach angle, steering operation, and timing for backing up, providing a route for safely and reliably entering a parking space.

[0426] 4. Implementation or assistance of automated driving

[0427] In Autonomous mode:

[0428] The device will then perform autonomous driving based on the generated optimal trajectory. In this mode, the device will automatically control steering, acceleration, and deceleration to complete the parking operation.

[0429] In driver assistance mode:

[0430] The device displays the generated optimal trajectory on an in-car monitor and assists the user in driving by following the displayed guidelines. The device also uses audio and visual alerts to support the user in taking appropriate actions at the appropriate time.

[0431] 5. New trajectory generation when leaving the warehouse

[0432] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates an optimal trajectory for leaving.

[0433] In Autonomous mode:

[0434] The terminal automatically drives the vehicle according to the generated departure trajectory and causes the vehicle to leave the parking lot.

[0435] In driver assistance mode:

[0436] The terminal displays the exit trajectory on the in-car monitor and assists the user in following the displayed guidelines. During exit, audio and visual alerts are also used to encourage appropriate operation.

[0437] Specific examples

[0438] Specific example of parking

[0439] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks the car in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly.

[0440] Specific example of shipping

[0441] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates a new optimal trajectory for leaving. The device then leaves the parking lot in autonomous driving mode or displays the trajectory in driver assistance mode, helping the user to follow it. This process allows the user to park and leave the parking lot with peace of mind.

[0442] The processing flow will be explained below.

[0443] Step 1:

[0444] The server acquires current surrounding images from a camera installed at the entrance of the parking lot, which are used to recognize the layout of parking spaces and the location of obstacles.

[0445] Step 2:

[0446] The server collects information on parking rules and available spaces from the database, which allows users to know the exact location of available spaces and important points to note when parking.

[0447] Step 3:

[0448] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[0449] Step 4:

[0450] The device sends the entered vehicle information to the server, which then receives all the data necessary for analysis.

[0451] Step 5:

[0452] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[0453] Step 6:

[0454] The server then transmits the generated optimal parking trajectory data to the terminal, allowing the terminal to learn the specific parking method.

[0455] Step 7:

[0456] When parking in autonomous driving mode based on the received parking trajectory, the device automatically performs steering, acceleration, and deceleration, thereby completing parking safely and efficiently.

[0457] Step 8:

[0458] When the device is in driver assistance mode, the generated parking trajectory is displayed on the in-car monitor. The user drives and parks according to this display. The device prompts the user to operate the vehicle at the appropriate time using audio and visual alerts.

[0459] Step 9:

[0460] After parking is complete, the terminal notifies the user, allowing the user to confirm that parking was completed successfully.

[0461] Step 10:

[0462] To start preparation for delivery, the user issues an instruction to the system, which makes the server ready to acquire new information.

[0463] Step 11:

[0464] The server then acquires images of the surrounding area again to obtain the latest information, thereby ensuring safety when the vehicle leaves the warehouse.

[0465] Step 12:

[0466] The server generates an optimal trajectory for the departure, calculating a new departure route and taking into account the location of obstacles and the movement of other vehicles.

[0467] Step 13:

[0468] The server then transmits the generated retrieval trajectory data to the terminal, which then retrieves the vehicle based on this information.

[0469] Step 14:

[0470] The terminal either drives automatically along the departure trajectory or displays the information on a monitor to assist the user in leaving the parking lot. In the autonomous driving mode, the terminal performs all operations automatically. In the driver assistance mode, the user operates the device themselves.

[0471] Step 15:

[0472] After the vehicle has left the parking lot, the terminal notifies the user, allowing the user to confirm that the vehicle has safely left the parking lot.

[0473] Example 1

[0474] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0475] Current parking assistance systems lack the precision and convenience required to achieve safe and efficient parking in tight parking spaces. The ability to quickly respond to changes in the surrounding environment is particularly important, and safety when exiting the parking space is equally important. Furthermore, the provision of interfaces and guidance to reduce the driver's burden is insufficient. The present invention aims to solve these problems.

[0476] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0477] In this invention, the server includes a means for acquiring surrounding conditions, a means for acquiring parking lot rule information, a means for inputting detailed vehicle information, a means for generating an optimal parking route based on the acquired images and vehicle information, and a means for automatically operating the vehicle or assisting the driver based on the generated parking route, thereby enabling the user to safely and efficiently park and exit even in a narrow parking space.

[0478] "Surrounding conditions" refers to information that indicates the state of objects and space around the vehicle.

[0479] "Parking lot rule information" refers to data regarding parking rules and available spaces that apply within a parking lot.

[0480] "Vehicle details" refers to specific data about the vehicle's dimensions and specifications (e.g., length, width, height, distance between axles, etc.).

[0481] An "optimal parking route" is the best calculated path for a vehicle to safely and efficiently enter a parking space.

[0482] "Autonomous operation" means that the system automatically controls the operation of the vehicle, including steering, acceleration, and deceleration.

[0483] "Means to assist the driver" refers to the system's ability to provide appropriate driving assistance to the driver in the form of voice or visual alerts, guidelines, etc.

[0484] An "optimal route for exit" is the calculated best path for a vehicle to safely exit a parking space.

[0485] An "in-vehicle display device" refers to a monitor or display device installed inside a vehicle that displays information to the driver.

[0486] The present invention is a system that assists drivers in parking and leaving narrow parking spaces safely and efficiently. The system includes a series of means for acquiring the surrounding situation, understanding parking lot rule information, inputting detailed vehicle information, generating optimal parking and leaving routes, and automatically operating the vehicle or assisting the driver based on the generated routes.

[0487] First, the server acquires the surrounding situation using a high-resolution camera installed at the entrance to the parking lot. This camera captures the surrounding image in real time and recognizes the layout of parking spaces and the location of obstacles using an object detection algorithm based on deep learning. The server also obtains information on parking lot rules and available spaces from a database. This enables accurate calculation of the parking route.

[0488] Next, the device provides a means for the user to input detailed vehicle information (e.g., length, width, height, axle distance, etc.). The input information is stored in a database and made available to the server. Based on this information, the server generates an optimal parking route. Specifically, it uses AI algorithms to calculate entry angles, steering operations, timing of backing, etc., and provides a route for safely and reliably entering a parking space. Specific technologies used include deep learning and reinforcement learning.

[0489] Based on the generated parking route, the device parks in either autonomous mode or driver-assisted mode. In autonomous mode, the device automatically controls steering, acceleration, and deceleration to complete the parking. In driver-assisted mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. Audio and visual alerts are used to help the user perform operations at the appropriate time.

[0490] After parking, when the user begins preparing to leave, the server again acquires information about the surrounding area and generates the optimal route for leaving. Both autonomous driving mode and driver assistance mode are available during leaving, allowing the user to leave the parking lot with peace of mind.

[0491] For example, you can instruct a generative AI model using specific prompts like the following:

[0492] 1. "Describe an automated driving assistance system for tight parking spaces."

[0493] 2. "How can I calculate the optimal parking route using vehicle and parking information?"

[0494] 3. "Please explain the difference between autonomous and driver assistance modes."

[0495] These prompts allow the system to properly obtain the necessary information and provide optimal parking and exit assistance.

[0496] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0497] Step 1: Acquire an image of the surroundings

[0498] The server acquires images of the surrounding area using a high-resolution camera installed at the entrance to the parking lot. This camera captures images in real time, and the server receives the image data. The acquired image data is analyzed using a deep learning algorithm to recognize the layout of parking spaces and the location of obstacles. The input is the camera's image data, and the output is the analyzed image data.

[0499] Step 2: Know the parking rules

[0500] The server retrieves rules information from a parking lot database. This database contains parking rules and available space information. For example, it checks whether a particular parking space is reserved or whether an obstacle exists. The input is a request for parking rules information, and the output is the parking rules and available space information.

[0501] Step 3: Enter vehicle information

[0502] The terminal provides a means for the user to input vehicle details (length, width, height, axle distance, etc.). The user enters these details through an input form. The input is the vehicle information entered by the user, and the output is the vehicle information stored in a database.

[0503] Step 4: Generate optimal parking trajectory

[0504] The server generates the optimal parking trajectory based on the surrounding image obtained in step 1, the parking lot rules acquired in step 2, and the detailed vehicle information entered in step 3. It uses AI algorithms (including deep learning and reinforcement learning) to calculate the approach angle, steering operation, timing of backing up, etc. The inputs are image data, parking lot rules, and vehicle information, and the output is the optimal parking route.

[0505] Step 5: Implementing or Assisting Autonomous Driving

[0506] In Autonomous mode:

[0507] The terminal performs autonomous driving based on the optimal parking trajectory received from the server. It controls the engine, steering, brakes, etc., and automatically parks the vehicle according to the instructed trajectory. The input is the optimal parking route, and the output is the completion of automatic parking.

[0508] In driver assistance mode:

[0509] The terminal displays the generated optimal trajectory on the monitor inside the vehicle and assists the user in parking by following the displayed guidelines. Guidelines for approach angles and steering operations are displayed on the navigation screen, and audio and visual alerts are used to prompt the user to perform operations at the appropriate time. The input is the optimal parking route, and the output is the completion of driver assistance.

[0510] Step 6: New trajectory generation for exit

[0511] When the user starts preparing to leave, the server acquires images of the surroundings again and generates an optimal trajectory for leaving. It calculates a safe route to leave, taking into account the position information of other vehicles and obstacles in the parking lot. The input is the latest surrounding images and information about the parking lot, and the output is the optimal route to leave.

[0512] Step 7: Conduct or assist with shipping

[0513] In Autonomous mode:

[0514] The terminal automatically drives the vehicle based on the generated departure trajectory and causes it to leave the parking lot. The input is the optimal departure route, and the output is the completion of automatic departure.

[0515] In driver assistance mode:

[0516] The terminal displays the departure trajectory on the in-vehicle monitor and assists the user in following the displayed guidelines. Audio and visual alerts are used to prompt appropriate operation. The input is the optimal departure route, and the output is completion of the driver's assistance with leaving the vehicle.

[0517] (Application example 1)

[0518] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0519] It is often difficult for drivers to safely and efficiently park in narrow parking spaces, especially in urban areas and crowded parking lots. Current parking assistance systems are insufficient, and further support is needed to enable drivers to park and exit safely. Another issue is the lack of technology that can identify available parking spaces and the location of obstacles in a parking lot in real time and provide the optimal parking method.

[0520] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0521] In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting the size and specifications of the vehicle, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for executing an artificial intelligence algorithm for recognizing the acquired images of the surroundings of the parking lot, and means for generating parking guidelines based on the recognized position information of vacant spaces and obstacles in the parking lot and displaying them on a screen. This enables the driver to park and exit a narrow parking space safely and efficiently with peace of mind.

[0522] The "means for acquiring images of the surroundings" refers to a means for collecting image data using a camera or a sensor in order to grasp the surrounding environment of the vehicle.

[0523] "Means for obtaining parking lot rules" refers to a means for obtaining information such as parking lot usage rules, layout, and available spaces from a database.

[0524] "Means for inputting vehicle size and specifications" refers to a means for a user to input detailed information such as the length, width, height, and axle distance of a vehicle.

[0525] The "means for generating the optimal parking trajectory" is a means for calculating and generating a safe and efficient parking route using an AI algorithm based on acquired image data and vehicle information.

[0526] "Means for automatically driving a vehicle or assisting a driver" refers to means including a function for automatically controlling the operation of a vehicle based on the generated parking trajectory, or a function for providing guidelines or voice guidance to support the driver when driving by himself.

[0527] "Means for executing artificial intelligence algorithms" refers to means for using AI technology to analyze images of the surrounding area of ​​the parking lot and identify the location of vacant spaces and obstacles.

[0528] "Means for generating parking guidelines and displaying them on the screen" refers to a means for generating guidelines for the driver to follow based on the recognized parking lot information and displaying them on a smartphone or in-car display.

[0529] The present invention is a system for assisting drivers in parking and leaving narrow parking spaces safely and efficiently. The system includes the following main means:

[0530] 1. A means of capturing images of the surroundings

[0531] The server collects image data using cameras and sensors to understand the surrounding environment of the parking lot. These cameras are installed at the entrance to the parking lot and around each parking space. The image data is acquired in real time and sent to the system.

[0532] 2. How to obtain parking rules

[0533] The server retrieves information about the parking lot, such as usage rules, layout, available spaces, etc. from a database. This information is regularly updated by the parking lot manager and provided to the server.

[0534] 3. A means of inputting vehicle size and specifications

[0535] Users can input detailed information such as vehicle length, width, height, and axle distance through the terminal, which is used to calculate the optimal parking trajectory.

[0536] 4. Means for generating optimal parking trajectories

[0537] The server uses an AI algorithm to calculate and generate a safe and efficient parking route based on the acquired image data and vehicle information. This algorithm determines the optimal trajectory by taking into account factors such as approach angle, steering operation, and timing of backing up.

[0538] 5. Means for autonomous driving or driver assistance of vehicles

[0539] The device provides a function to automatically control vehicle operation based on the generated parking trajectory. It also supports the driver by providing parking guidelines and voice guidance when driving independently. This function displays parking guidelines on a smartphone or in-car display to help the driver park according to them.

[0540] 6. Means for running artificial intelligence algorithms

[0541] The server uses AI technology to analyze images of the surrounding parking lot and identify the location of available spaces and obstacles. This AI algorithm uses a deep learning model to suggest the optimal parking strategy based on the results of the image analysis.

[0542] 7. A method for generating parking guidelines and displaying them on the screen

[0543] The device generates guidelines for the driver to follow based on the recognized parking information, displays them on a smartphone or in-car display, and uses voice synthesis to provide driving instructions at the appropriate time.

[0544] Specific examples

[0545] For example, a user arrives at a small parking lot in central Tokyo and launches a parking support app on their smartphone. The app scans the entire parking lot using a camera and calculates the optimal parking route using an AI algorithm. The user parks their car according to the guidelines displayed on the smartphone screen and uses voice guidance to perform the appropriate operations, completing the parking process with peace of mind. An example of a prompt that can be applied at this time is, "Please recognize the parking lot layout and identify the location of available spaces and obstacles."

[0546] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0547] Step 1:

[0548] Input: Real-time image data from a parking lot camera.

[0549] Processing: The server uses the parking lot cameras to capture images of the surrounding area, including footage of the parking space and any obstacles in the area.

[0550] Output: Captured parking lot image data.

[0551] Step 2:

[0552] Input: Parking rules, available space information, and image data.

[0553] Processing: The server collects information about parking rules and available spaces from a database and compares it with the captured image data, for example, identifying no-parking areas and the location of obstacles.

[0554] Output: Parking lot layout information as a result of image processing (available parking areas, empty spaces, obstacle locations).

[0555] Step 3:

[0556] Input: Vehicle size and specifications (length, width, height, axle distance).

[0557] Processing: The user uses a device (smartphone) to input vehicle information, which is then sent to the server.

[0558] Output: Transmitted vehicle information data.

[0559] Step 4:

[0560] Input: Parking lot layout information, vehicle information.

[0561] Processing: The server uses AI algorithms to generate the optimal parking trajectory based on the input vehicle information and parking lot layout information, for example, calculating the parking space entry angle and backing timing.

[0562] Output: Generated parking trajectory data.

[0563] Step 5:

[0564] Input: Parking trajectory data.

[0565] Processing: Based on the generated optimal parking trajectory, the device generates parking guidelines to assist the driver and displays them on a smartphone or in-car display. It also uses the voice guidance function to provide driving instructions at the appropriate time.

[0566] Output: Parking guideline display data and voice guidance data provided to the user.

[0567] Step 6:

[0568] Input: User action to prepare for delivery.

[0569] Processing: When the user begins preparing to leave, the server again captures images of the surroundings and uses AI algorithms to generate an optimal exit trajectory, taking into account any new obstacles or changing parking lot conditions.

[0570] Output: Generated outbound trajectory data.

[0571] Step 7:

[0572] Input: Outbound trajectory data.

[0573] Processing: Based on the generated departure trajectory, the terminal generates departure guidelines to assist the driver and displays them on a smartphone or in-car display. It also uses a voice guidance function to provide departure instructions at the appropriate time.

[0574] Output: Display data and voice guidance data for delivery guidelines provided to the user.

[0575] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0576] The present invention is a system that assists drivers in parking in tight spaces safely and efficiently, and further combines it with an emotion engine that recognizes the user's emotions. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automatic driving or driver assistance, as well as analyzing the user's emotions.

[0577] Explanation of program processing

[0578] 1. Acquiring surrounding images and understanding parking lot rules

[0579] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[0580] 2. Enter your car information

[0581] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[0582] 3. Parking Trajectory Generation

[0583] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[0584] 4. Implementation or assistance of automated driving

[0585] In Autonomous mode:

[0586] The device will then perform autonomous driving based on the generated optimal trajectory. In this mode, the device will automatically control steering, acceleration, and deceleration to complete the parking operation.

[0587] In driver assistance mode:

[0588] The device displays the generated optimal trajectory on an in-car monitor and assists the user in driving by following the displayed guidelines. The device also prompts the user to take action at the appropriate time using audio and visual alerts.

[0589] 5. User Emotion Recognition

[0590] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling stressed or anxious, that emotion is conveyed to the system.

[0591] 6. Emotion-based system adjustment

[0592] The server adjusts the system's behavior based on the recognized user's emotions.

[0593] If the user is feeling stressed, the device will play relaxing music and gently guide them through operations.

[0594] If the user feels anxious or nervous, the device will preferentially select the driving assistance mode and provide detailed guidance on the parking trajectory.

[0595] 7. New trajectory generation when leaving the warehouse

[0596] After parking, when the user starts preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. Based on the input information, it generates the optimal trajectory for leaving the parking lot.

[0597] 8. Conducting or assisting with deliveries

[0598] In Autonomous mode:

[0599] The terminal automatically drives the vehicle according to the generated departure trajectory and causes the vehicle to leave the parking lot.

[0600] In driver assistance mode:

[0601] The terminal displays the exit trajectory on the in-car monitor and guides the user through the exit process by following the displayed guidelines, using audio and visual alerts to encourage appropriate operation.

[0602] Specific examples

[0603] Specific example of parking

[0604] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly. The device also recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[0605] Specific example of shipping

[0606] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates a new optimal trajectory for leaving. The device then either leaves the vehicle in autonomous driving mode or displays the trajectory in driver assistance mode and helps the user leave the vehicle accordingly. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[0607] The processing flow will be explained below.

[0608] Step 1:

[0609] The server acquires current surrounding images from a camera installed at the entrance of the parking lot, which are used to recognize the layout of parking spaces and the location of obstacles.

[0610] Step 2:

[0611] The server collects information on parking rules and available spaces from the database, which allows users to know the exact location of available spaces and important points to note when parking.

[0612] Step 3:

[0613] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[0614] Step 4:

[0615] The device sends the entered vehicle information to the server, which then receives all the data necessary for analysis.

[0616] Step 5:

[0617] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[0618] Step 6:

[0619] The server then transmits the generated optimal parking trajectory data to the terminal, allowing the terminal to learn the specific parking method.

[0620] Step 7:

[0621] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, it uses facial expression analysis and voice analysis technology to detect whether the user is feeling stressed or anxious.

[0622] Step 8:

[0623] When the emotion engine recognizes emotions such as stress or anxiety, the device automatically adjusts system behavior, including playing relaxing music and providing more detailed operating instructions.

[0624] Step 9:

[0625] When parking in autonomous driving mode based on the received parking trajectory, the device automatically performs steering, acceleration, and deceleration, thereby completing parking safely and efficiently.

[0626] Step 10:

[0627] When the device is in driver assistance mode, the generated parking trajectory is displayed on the in-car monitor. The user drives and parks according to this display. The device prompts the user to operate the vehicle at the appropriate time using audio and visual alerts.

[0628] Step 11:

[0629] After parking is complete, the terminal notifies the user, allowing the user to confirm that parking was completed successfully.

[0630] Step 12:

[0631] To start preparation for delivery, the user issues an instruction to the system, which makes the server ready to acquire new information.

[0632] Step 13:

[0633] The server then acquires images of the surrounding area again to obtain the latest information, thereby ensuring safety when the vehicle leaves the warehouse.

[0634] Step 14:

[0635] The server generates an optimal trajectory for the departure, calculating a new departure route and taking into account the location of obstacles and the movement of other vehicles.

[0636] Step 15:

[0637] The server then transmits the generated retrieval trajectory data to the terminal, which then retrieves the vehicle based on this information.

[0638] Step 16:

[0639] The terminal either drives automatically along the departure trajectory or displays the information on a monitor to assist the user in leaving the parking lot. In the autonomous driving mode, the terminal performs all operations automatically. In the driver assistance mode, the user operates the device themselves.

[0640] Step 17:

[0641] After the vehicle has left the parking lot, the terminal notifies the user, allowing the user to confirm that the vehicle has safely left the parking lot.

[0642] Example 2

[0643] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0644] Parking safely and efficiently in tight parking spaces is a challenging task for many drivers. The stress and anxiety drivers feel during parking also impact safety. However, current automated driving and driver assistance systems lack the ability to recognize user emotions and adapt to the user's psychological state during the parking process. Therefore, there is a need for a parking assistance system that can adjust its behavior based on the user's emotions.

[0645] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0646] In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting the size and specifications of the vehicle, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for recognizing the user's emotions, and means for adjusting the system operation based on the recognized user's emotions, thereby making it possible to provide optimal parking support while taking the user's emotions into consideration.

[0647] "Means for acquiring images of the surrounding area" refers to technology that collects real-time images of the surrounding area using cameras and sensors installed at the entrance to the parking lot.

[0648] "Means for obtaining parking lot rules" refers to technology for obtaining traffic rules and available space information within a parking lot from a database.

[0649] "Means for inputting vehicle dimensions and specifications" refers to technology that provides an interface for users to input the dimensions and specifications (e.g., length, width, height, axle distance, etc.) of their vehicle.

[0650] "Means for generating optimal parking trajectories" refers to technology that uses AI algorithms to calculate entry angles, steering operation, timing of backing up, etc. based on acquired surrounding images and vehicle information to generate optimal parking trajectories.

[0651] "Means for autonomously driving a vehicle or assisting the driver" refers to technology that automatically controls the vehicle's steering, acceleration, and deceleration based on the generated parking trajectory, or technology that displays the trajectory on an in-vehicle monitor and assists the driver with audio and visual alerts.

[0652] "Means for recognizing user emotions" refers to an emotion engine that analyzes emotions such as stress and anxiety from the user's facial expressions and voice.

[0653] "Means for adjusting system behavior" refers to functions that change the behavior of the system based on the recognized user's emotions, such as playing relaxing music or providing gentler operating guidance.

[0654] This invention combines a system that assists drivers in parking in tight spaces safely and efficiently with an emotion engine that recognizes the user's emotions. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automatic driving or driver assistance, as well as analyzing the user's emotions.

[0655] The detailed system configuration uses the following hardware and software:

[0656] Acquiring surrounding images and understanding parking rules

[0657] The server obtains current surrounding images from a camera installed at the entrance to the parking lot. This high-resolution camera can clearly recognize obstacles and available spaces within the parking lot. Image analysis is performed in real time using image recognition software such as OpenCV. At the same time, the server collects information on parking lot rules and available spaces from a database. This database records traffic rules within the parking lot and space usage status.

[0658] Enter your car information

[0659] The device provides an interface for the user to input vehicle dimensions and specifications (e.g., length, width, height, and axle distance). This interface may take the form of a smartphone app or an in-vehicle display. The input information is stored on the device and sent to a server to avoid having to enter it again.

[0660] Parking trajectory generation

[0661] The server performs data analysis based on the acquired surrounding images and vehicle information. AI algorithms (e.g., TensorFlow and PyTorch) are used for the analysis, calculating factors such as approach angle, steering operation, and timing of backing up, and generating the optimal parking trajectory. The generated trajectory information is sent to the device.

[0662] Implementation or assistance of automated driving

[0663] In autonomous driving mode, the device automatically drives the vehicle based on the generated optimal trajectory. In this mode, the device automatically controls steering, acceleration, and deceleration, and completes parking using dedicated autonomous driving software (e.g., Autoware).

[0664] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. The device uses audio and visual alerts to prompt the user to take action at the appropriate time.

[0665] User Emotion Recognition

[0666] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. Emotion analysis utilizes facial and voice analysis software such as Microsoft Azure Face API and Google Speech-to-Text. For example, if the user is feeling stressed or anxious, that emotion is conveyed to the system.

[0667] Emotion-based system adjustment

[0668] The server adjusts the system's behavior based on the user's recognized emotions. Specifically, if the user feels stressed, the device will play relaxing music (e.g., using Spotify or Apple Music APIs) and provide gentle guidance on operations. If the user feels anxious or tense, the device will prioritize driving assistance mode and provide detailed guidelines.

[0669] New trajectory generation when leaving the warehouse

[0670] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area to ascertain the latest situation. Based on the input information, an AI algorithm is used to generate the optimal trajectory for leaving the parking lot and send it to the terminal.

[0671] Conducting or assisting with shipping

[0672] In autonomous driving mode, the device automatically drives the vehicle according to the generated exit trajectory and leaves the parking lot. In driver assistance mode, the device displays the exit trajectory on the in-car monitor and helps the user leave the parking lot by following the displayed guidelines. Audio and visual alerts are also used during this process.

[0673] Specific examples

[0674] As a concrete example of parking, when a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device that receives this information either parks in autonomous driving mode or displays the trajectory on a monitor in driver assistance mode, helping the user park accordingly. At the same time, the device recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[0675] Example prompts to input to the generative AI model:

[0676] "A user arrives at a parking lot, and the system automatically finds a parking space and assists the user. If the user is emotionally stressed, how will the system respond?"

[0677] For example, when the user prepares to leave the parking lot after parking, the server acquires images of the surrounding area again and generates a new optimal trajectory for leaving. The device then either leaves the parking lot in autonomous driving mode or displays the trajectory in driver assistance mode and assists the user in leaving the parking lot. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[0678] Example prompts to input to the generative AI model:

[0679] "When the user attempts to leave the parking lot after parking, please explain how the system will assist the user in leaving the parking lot and how it will respond depending on the user's emotions."

[0680] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0681] Step 1: Acquire surrounding images and understand parking rules

[0682] The server obtains real-time images of the surrounding area from a camera installed at the entrance to the parking lot. The camera image is used as input, and the obtained image data is obtained as output. This image data is analyzed to recognize the layout of parking spaces and the location of obstacles. Image recognition software such as OpenCV is used for image analysis. At the same time, the server obtains information on parking lot rules and available spaces from a database. The information registered by the user at the parking lot is used as input, and the obtained parking lot rule data is obtained as output.

[0683] Specific behavior:

[0684] The server receives the video stream from the camera.

[0685] The server performs image analysis and recognizes parking spaces and obstacles.

[0686] The server retrieves the parking rules from the database.

[0687] Step 2: Enter your vehicle information

[0688] The terminal allows the user to input the vehicle size and specifications (length, width, height, axle distance, etc.). The user's input data is used as input, and stored vehicle information is obtained as output. This information is saved on the terminal and later sent to the server. The stored data is reused to avoid having to enter the information again.

[0689] Specific behavior:

[0690] The device displays the vehicle information input screen.

[0691] The user inputs the vehicle size and specifications.

[0692] The device saves the input data and sends it to the server.

[0693] Step 3: Generate parking trajectories

[0694] The server performs data analysis based on the acquired surrounding images and the user's vehicle information. Surrounding image data and vehicle information data are used as input, and parking trajectory data is generated as output. Here, AI algorithms (e.g., TensorFlow and PyTorch) are used to calculate factors such as approach angle, steering operation, and timing of backing up, and generate the optimal parking trajectory.

[0695] Specific behavior:

[0696] The server acquires surrounding images and vehicle information.

[0697] The server analyzes using an AI algorithm.

[0698] The server generates the optimal parking trajectory and sends it to the device.

[0699] Step 4: Implementing or Assisting Autonomous Driving

[0700] In autonomous driving mode, the device performs autonomous driving based on the generated optimal trajectory. Parking trajectory data is used as input, and the parking completion status is obtained as output. Dedicated autonomous driving software (e.g., Autoware) is used.

[0701] Specific behavior:

[0702] The terminal receives the optimal orbit data.

[0703] The device begins autonomous driving using Autoware.

[0704] The device controls the steering wheel, accelerator, and brakes.

[0705] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user to drive according to the guideline. The input is the parking trajectory data, and the output is driving assistance for the user. The device uses audio and visual alerts.

[0706] Specific behavior:

[0707] The terminal displays the optimal orbit data on the monitor.

[0708] The user drives according to the guidelines.

[0709] The device assists the user with voice alerts.

[0710] Step 5: Recognizing User Emotions

[0711] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. The user's facial expression and voice data are used as input, and the recognized emotion data is obtained as output. Emotion analysis uses Microsoft Azure Face API and Google Speech-to-Text, among others.

[0712] Specific behavior:

[0713] The device uses a camera and microphone to collect the user's facial expressions and voice.

[0714] The device analyzes using an emotion engine.

[0715] The device sends the analysis results to the server.

[0716] Step 6: Adjusting the system based on emotions

[0717] The server adjusts the system's behavior based on the recognized user emotion. Emotional data is used as input and the adjusted system behavior is obtained as output. If the user is feeling stressed, the device will play relaxing music (using Spotify or Apple Music APIs) or provide gentle guidance.

[0718] Specific behavior:

[0719] The server processes the emotion data.

[0720] The device will select and play the appropriate music.

[0721] The device provides a friendly audio guide.

[0722] Step 7: New trajectory generation for exit

[0723] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. The surrounding image data is used as input, and the optimal trajectory data for leaving is obtained as output. The optimal trajectory for leaving is generated using an AI algorithm and sent to the terminal.

[0724] Specific behavior:

[0725] The user begins preparation for delivery.

[0726] The server acquires surrounding images from the camera.

[0727] The server generates the optimal trajectory for delivery and sends it to the terminal.

[0728] Step 8: Perform or assist with shipping

[0729] In the automatic driving mode, the terminal automatically drives according to the generated retrieval trajectory. The retrieval trajectory data is used as input, and the retrieval completion status is obtained as output.

[0730] Specific behavior:

[0731] The terminal receives the delivery trajectory data.

[0732] The terminal will execute the retrieval in automatic operation mode.

[0733] In driver assistance mode, the device displays the exit trajectory on the in-vehicle monitor and assists the user in following the guidelines when leaving the vehicle. The exit trajectory data is used as input, and driving assistance is provided to the user as output. Audio and visual alerts encourage appropriate operation.

[0734] Specific behavior:

[0735] The terminal displays the delivery trajectory data on the monitor.

[0736] The user issues goods according to the guidelines.

[0737] The device will prompt you with an audio alert.

[0738] (Application example 2)

[0739] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0740] Modern parking lots often have narrow spaces and complex layouts, making parking difficult for drivers. The stress and anxiety drivers feel when parking can negatively impact driving safety and efficiency. Even in these situations, there is a demand for systems that support safe and efficient parking. Furthermore, systems must provide support not only when parking but also when exiting the parking lot safely and efficiently.

[0741] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting vehicle size and specifications, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for analyzing user emotions, and means for adjusting system operation based on the user emotions. This enables the driver to park safely and efficiently, and by providing support according to the driver's emotions, it becomes possible to reduce stress and anxiety.

[0742] The "surrounding image" is an image acquired using a device such as a camera to recognize the layout of the parking lot and the location of obstacles.

[0743] "Parking rules" are rules and instructions regarding parking spaces, prohibited areas, and how to pass through a parking lot.

[0744] "Vehicle size and specifications" refers to information that refers to the physical dimensions and characteristics of a vehicle, such as length, width, height, and axle distance.

[0745] A "parking trajectory" is a planned path for a vehicle to park in a parking space, including the optimal approach angle, steering operation, and timing of backing up.

[0746] "Autonomous driving" is the ability of a vehicle to perform driving operations autonomously without human intervention.

[0747] "Supporting the driver" means providing assistance by providing guidelines, alerts, etc. when the driver parks or leaves the vehicle by themselves.

[0748] "User's emotions" refer to psychological states such as stress, anxiety, joy, etc. that are recognized from the user's facial expressions, voice, etc.

[0749] "Analyzing emotions" refers to the process of analyzing the user's facial expressions and voice to identify their emotions.

[0750] "Adjusting the system's behavior" means changing the system's functions and behavior in response to the user's emotions, for example, by playing relaxing music or providing gentle guidance.

[0751] This invention relates to a system that assists drivers in parking in narrow parking spaces safely and efficiently. The system includes multiple means for capturing images of the surrounding area, understanding parking rules, inputting vehicle information, generating an optimal parking trajectory, and performing automated driving or driver assistance and user emotion analysis.

[0752] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[0753] The terminal prompts the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.). This information is stored in the terminal, eliminating the need to input it again. The server generates the optimal parking trajectory based on the captured surrounding images and vehicle information. The server uses an AI algorithm (e.g., TensorFlow) to calculate factors such as approach angle, steering operation, and timing for backing up, and generates the optimal parking trajectory.

[0754] In autonomous driving mode, the device drives autonomously based on the generated optimal trajectory. In this mode, the device automatically controls steering, acceleration, and deceleration to complete parking. In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. The device uses audio and visual alerts to prompt the user to operate at the appropriate time.

[0755] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling stressed or anxious, that emotion is communicated to the system. The server adjusts the system's operation based on the recognized user emotion. If the user is feeling stressed, the device may play relaxing music or gently guide the user through the operation. If the user is feeling anxious or tense, the device may prioritize driving assistance mode and provide detailed guidance on the parking trajectory.

[0756] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area and ascertains the latest situation. Based on the input information, it generates an optimal trajectory for leaving. In autonomous driving mode, the device automatically drives the vehicle according to the generated trajectory and leaves the parking lot. In driver assistance mode, the device displays the trajectory on the in-car monitor and assists the user in leaving the parking lot by following the displayed guidelines. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[0757] Specific examples

[0758] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly. The device also recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[0759] Prompt Sentence Examples

[0760] "A user is trying to park a car in a tight parking space. Implement a system that recognizes whether the user is stressed or not from their facial expression, and generates an optimal parking trajectory to assist the user in parking in real time."

[0761] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0762] Step 1:

[0763] The server obtains current surrounding images from a camera installed at the entrance to the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. The video data obtained from the camera (input) is used to identify the location of vacant spaces in the parking lot and obstacles (example of data processing: object detection using image processing algorithms). The output becomes map information of the parking lot.

[0764] Step 2:

[0765] The server collects information about parking rules and available spaces from a database. This information (input) includes rules and instructions about permitted parking spaces, prohibited areas, and how to navigate. The output is the optimal parking location that complies with the permitted parking areas and rules. This is achieved by executing a database query to obtain the required information.

[0766] Step 3:

[0767] The terminal allows the user to input the vehicle size and specifications (e.g. length, width, height, axle distance, etc.). This information (input) is stored in a vehicle database. The (output) is the complete vehicle dimensions, held in a reusable format. Input is received through a user interface.

[0768] Step 4:

[0769] The server generates an optimal parking trajectory based on the acquired surrounding images and vehicle information. It uses an AI algorithm (e.g., TensorFlow) to calculate the approach angle, steering operation, timing of backing up, etc. (example of data calculation: application of a trajectory prediction model using a neural network) and generates the generated parking trajectory (output).

[0770] Step 5:

[0771] In autonomous driving mode, the device performs autonomous driving based on the generated optimal trajectory. It automatically controls steering, acceleration, and deceleration (vehicle operation using a control algorithm) and completes parking (output). The input is optimal trajectory information. Sensor information is fed back in real time to control the vehicle's movement.

[0772] Step 6:

[0773] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. It uses audio and visual alerts to prompt the user to operate the vehicle at the appropriate time (input is the generated parking trajectory information, output is guideline display and alerts) to complete parking. It generates audio and visual guides and displays them on the user interface.

[0774] Step 7:

[0775] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. It analyzes the user's facial images and voice data (input) (example of data processing: facial expression and voice analysis using machine learning models) and identifies the user's state of stress or anxiety (output). It then identifies the user's emotional state based on the results of the data analysis.

[0776] Step 8:

[0777] The server adjusts the system's behavior based on the recognized user emotion. If the user feels stressed, the device plays relaxing music (input: emotion recognition result, output: music playback) and gently guides the user through the operation. If the user feels anxious or tense, the device prioritizes driving assistance mode and provides more detailed guidance (output), adjusts system settings, and provides appropriate feedback.

[0778] Step 9:

[0779] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. The acquired image data (input) is used to re-recognize the surrounding environment (example of data processing: re-application of object detection algorithm) and generate the optimal trajectory for leaving (output). The optimal exit route is calculated through data analysis.

[0780] Step 10:

[0781] In autonomous driving mode, the terminal automatically drives the vehicle according to the generated departure trajectory and leaves the parking lot. It performs autonomous driving control based on the departure trajectory information (input) and optimizes the vehicle's movement (output). It executes departure operations including real-time feedback control.

[0782] Step 11:

[0783] In driver assistance mode, the device displays the exit trajectory on the in-car monitor and assists the user in following the displayed guidelines to complete the exit. Audio and visual alerts are used to prompt the user to take the appropriate action and complete the exit (input is the exit trajectory information, output is the guideline display and alerts). The device makes full use of the user interface to provide appropriate guidance and feedback.

[0784] Example prompt: "A user is trying to park a car in a narrow parking space. Implement a system that recognizes whether the user is stressed from their facial expression and generates an optimal parking trajectory to assist the user in parking in real time."

[0785] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0786] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0787] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0788] [Third embodiment]

[0789] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0790] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0791] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0792] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0793] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0794] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0795] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0796] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0797] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0798] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0799] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0800] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0801] The present invention is a system that assists drivers in parking in tight spaces safely and efficiently. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automated driving or driver assistance.

[0802] Explanation of program processing

[0803] 1. Acquiring surrounding images and understanding parking lot rules

[0804] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[0805] 2. Enter your car information

[0806] The terminal provides a means for the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) so that the system can obtain accurate vehicle information to use in calculating the optimal parking trajectory.

[0807] 3. Parking Trajectory Generation

[0808] The server uses the captured images of the surroundings and vehicle information to generate the optimal parking trajectory, using an AI algorithm to calculate factors such as approach angle, steering operation, and timing for backing up, providing a route for safely and reliably entering a parking space.

[0809] 4. Implementation or assistance of automated driving

[0810] In Autonomous mode:

[0811] The device will then perform autonomous driving based on the generated optimal trajectory. In this mode, the device will automatically control steering, acceleration, and deceleration to complete the parking operation.

[0812] In driver assistance mode:

[0813] The device displays the generated optimal trajectory on an in-car monitor and assists the user in driving by following the displayed guidelines. The device also uses audio and visual alerts to support the user in taking appropriate actions at the appropriate time.

[0814] 5. New trajectory generation when leaving the warehouse

[0815] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates an optimal trajectory for leaving.

[0816] In Autonomous mode:

[0817] The terminal automatically drives the vehicle according to the generated departure trajectory and causes the vehicle to leave the parking lot.

[0818] In driver assistance mode:

[0819] The terminal displays the exit trajectory on the in-car monitor and assists the user in following the displayed guidelines. During exit, audio and visual alerts are also used to encourage appropriate operation.

[0820] Specific examples

[0821] Specific example of parking

[0822] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks the car in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly.

[0823] Specific example of shipping

[0824] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates a new optimal trajectory for leaving. The device then leaves the parking lot in autonomous driving mode or displays the trajectory in driver assistance mode, helping the user to follow it. This process allows the user to park and leave the parking lot with peace of mind.

[0825] The processing flow will be explained below.

[0826] Step 1:

[0827] The server acquires current surrounding images from a camera installed at the entrance of the parking lot, which are used to recognize the layout of parking spaces and the location of obstacles.

[0828] Step 2:

[0829] The server collects information on parking rules and available spaces from the database, which allows users to know the exact location of available spaces and important points to note when parking.

[0830] Step 3:

[0831] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[0832] Step 4:

[0833] The device sends the entered vehicle information to the server, which then receives all the data necessary for analysis.

[0834] Step 5:

[0835] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[0836] Step 6:

[0837] The server then transmits the generated optimal parking trajectory data to the terminal, allowing the terminal to learn the specific parking method.

[0838] Step 7:

[0839] When parking in autonomous driving mode based on the received parking trajectory, the device automatically performs steering, acceleration, and deceleration, thereby completing parking safely and efficiently.

[0840] Step 8:

[0841] When the device is in driver assistance mode, the generated parking trajectory is displayed on the in-car monitor. The user drives and parks according to this display. The device prompts the user to operate the vehicle at the appropriate time using audio and visual alerts.

[0842] Step 9:

[0843] After parking is complete, the terminal notifies the user, allowing the user to confirm that parking was completed successfully.

[0844] Step 10:

[0845] To start preparation for delivery, the user issues an instruction to the system, which makes the server ready to acquire new information.

[0846] Step 11:

[0847] The server then acquires images of the surrounding area again to obtain the latest information, thereby ensuring safety when the vehicle leaves the warehouse.

[0848] Step 12:

[0849] The server generates an optimal trajectory for the departure, calculating a new departure route and taking into account the location of obstacles and the movement of other vehicles.

[0850] Step 13:

[0851] The server then transmits the generated retrieval trajectory data to the terminal, which then retrieves the vehicle based on this information.

[0852] Step 14:

[0853] The terminal either drives automatically along the departure trajectory or displays the information on a monitor to assist the user in leaving the parking lot. In the autonomous driving mode, the terminal performs all operations automatically. In the driver assistance mode, the user operates the device themselves.

[0854] Step 15:

[0855] After the vehicle has left the parking lot, the terminal notifies the user, allowing the user to confirm that the vehicle has safely left the parking lot.

[0856] Example 1

[0857] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0858] Current parking assistance systems lack the precision and convenience required to achieve safe and efficient parking in tight parking spaces. The ability to quickly respond to changes in the surrounding environment is particularly important, and safety when exiting the parking space is equally important. Furthermore, the provision of interfaces and guidance to reduce the driver's burden is insufficient. The present invention aims to solve these problems.

[0859] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0860] In this invention, the server includes a means for acquiring surrounding conditions, a means for acquiring parking lot rule information, a means for inputting detailed vehicle information, a means for generating an optimal parking route based on the acquired images and vehicle information, and a means for automatically operating the vehicle or assisting the driver based on the generated parking route, thereby enabling the user to safely and efficiently park and exit even in a narrow parking space.

[0861] "Surrounding conditions" refers to information that indicates the state of objects and space around the vehicle.

[0862] "Parking lot rule information" refers to data regarding parking rules and available spaces that apply within a parking lot.

[0863] "Vehicle details" refers to specific data about the vehicle's dimensions and specifications (e.g., length, width, height, distance between axles, etc.).

[0864] An "optimal parking route" is the best calculated path for a vehicle to safely and efficiently enter a parking space.

[0865] "Autonomous operation" means that the system automatically controls the operation of the vehicle, including steering, acceleration, and deceleration.

[0866] "Means to assist the driver" refers to the system's ability to provide appropriate driving assistance to the driver in the form of voice or visual alerts, guidelines, etc.

[0867] An "optimal route for exit" is the calculated best path for a vehicle to safely exit a parking space.

[0868] An "in-vehicle display device" refers to a monitor or display device installed inside a vehicle that displays information to the driver.

[0869] The present invention is a system that assists drivers in parking and leaving narrow parking spaces safely and efficiently. The system includes a series of means for acquiring the surrounding situation, understanding parking lot rule information, inputting detailed vehicle information, generating optimal parking and leaving routes, and automatically operating the vehicle or assisting the driver based on the generated routes.

[0870] First, the server acquires the surrounding situation using a high-resolution camera installed at the entrance to the parking lot. This camera captures the surrounding image in real time and recognizes the layout of parking spaces and the location of obstacles using an object detection algorithm based on deep learning. The server also obtains information on parking lot rules and available spaces from a database. This enables accurate calculation of the parking route.

[0871] Next, the device provides a means for the user to input detailed vehicle information (e.g., length, width, height, axle distance, etc.). The input information is stored in a database and made available to the server. Based on this information, the server generates an optimal parking route. Specifically, it uses AI algorithms to calculate entry angles, steering operations, timing of backing, etc., and provides a route for safely and reliably entering a parking space. Specific technologies used include deep learning and reinforcement learning.

[0872] Based on the generated parking route, the device parks in either autonomous mode or driver-assisted mode. In autonomous mode, the device automatically controls steering, acceleration, and deceleration to complete the parking. In driver-assisted mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. Audio and visual alerts are used to help the user perform operations at the appropriate time.

[0873] After parking, when the user begins preparing to leave, the server again acquires information about the surrounding area and generates the optimal route for leaving. Both autonomous driving mode and driver assistance mode are available during leaving, allowing the user to leave the parking lot with peace of mind.

[0874] For example, you can instruct a generative AI model using specific prompts like the following:

[0875] 1. "Describe an automated driving assistance system for tight parking spaces."

[0876] 2. "How can I calculate the optimal parking route using vehicle and parking information?"

[0877] 3. "Please explain the difference between autonomous and driver assistance modes."

[0878] These prompts allow the system to properly obtain the necessary information and provide optimal parking and exit assistance.

[0879] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0880] Step 1: Acquire an image of the surroundings

[0881] The server acquires images of the surrounding area using a high-resolution camera installed at the entrance to the parking lot. This camera captures images in real time, and the server receives the image data. The acquired image data is analyzed using a deep learning algorithm to recognize the layout of parking spaces and the location of obstacles. The input is the camera's image data, and the output is the analyzed image data.

[0882] Step 2: Know the parking rules

[0883] The server retrieves rules information from a parking lot database. This database contains parking rules and available space information. For example, it checks whether a particular parking space is reserved or whether an obstacle exists. The input is a request for parking rules information, and the output is the parking rules and available space information.

[0884] Step 3: Enter vehicle information

[0885] The terminal provides a means for the user to input vehicle details (length, width, height, axle distance, etc.). The user enters these details through an input form. The input is the vehicle information entered by the user, and the output is the vehicle information stored in a database.

[0886] Step 4: Generate optimal parking trajectory

[0887] The server generates the optimal parking trajectory based on the surrounding image obtained in step 1, the parking lot rules acquired in step 2, and the detailed vehicle information entered in step 3. It uses AI algorithms (including deep learning and reinforcement learning) to calculate the approach angle, steering operation, timing of backing up, etc. The inputs are image data, parking lot rules, and vehicle information, and the output is the optimal parking route.

[0888] Step 5: Implementing or Assisting Autonomous Driving

[0889] In Autonomous mode:

[0890] The terminal performs autonomous driving based on the optimal parking trajectory received from the server. It controls the engine, steering, brakes, etc., and automatically parks the vehicle according to the instructed trajectory. The input is the optimal parking route, and the output is the completion of automatic parking.

[0891] In driver assistance mode:

[0892] The terminal displays the generated optimal trajectory on the monitor inside the vehicle and assists the user in parking by following the displayed guidelines. Guidelines for approach angles and steering operations are displayed on the navigation screen, and audio and visual alerts are used to prompt the user to perform operations at the appropriate time. The input is the optimal parking route, and the output is the completion of driver assistance.

[0893] Step 6: New trajectory generation for exit

[0894] When the user starts preparing to leave, the server acquires images of the surroundings again and generates an optimal trajectory for leaving. It calculates a safe route to leave, taking into account the position information of other vehicles and obstacles in the parking lot. The input is the latest surrounding images and information about the parking lot, and the output is the optimal route to leave.

[0895] Step 7: Conduct or assist with shipping

[0896] In Autonomous mode:

[0897] The terminal automatically drives the vehicle based on the generated departure trajectory and causes it to leave the parking lot. The input is the optimal departure route, and the output is the completion of automatic departure.

[0898] In driver assistance mode:

[0899] The terminal displays the departure trajectory on the in-vehicle monitor and assists the user in following the displayed guidelines. Audio and visual alerts are used to prompt appropriate operation. The input is the optimal departure route, and the output is completion of the driver's assistance with leaving the vehicle.

[0900] (Application example 1)

[0901] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0902] It is often difficult for drivers to safely and efficiently park in narrow parking spaces, especially in urban areas and crowded parking lots. Current parking assistance systems are insufficient, and further support is needed to enable drivers to park and exit safely. Another issue is the lack of technology that can identify available parking spaces and the location of obstacles in a parking lot in real time and provide the optimal parking method.

[0903] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0904] In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting the size and specifications of the vehicle, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for executing an artificial intelligence algorithm for recognizing the acquired images of the surroundings of the parking lot, and means for generating parking guidelines based on the recognized position information of vacant spaces and obstacles in the parking lot and displaying them on a screen. This enables the driver to park and exit a narrow parking space safely and efficiently with peace of mind.

[0905] The "means for acquiring images of the surroundings" refers to a means for collecting image data using a camera or a sensor in order to grasp the surrounding environment of the vehicle.

[0906] "Means for obtaining parking lot rules" refers to a means for obtaining information such as parking lot usage rules, layout, and available spaces from a database.

[0907] "Means for inputting vehicle size and specifications" refers to a means for a user to input detailed information such as the length, width, height, and axle distance of a vehicle.

[0908] The "means for generating the optimal parking trajectory" is a means for calculating and generating a safe and efficient parking route using an AI algorithm based on acquired image data and vehicle information.

[0909] "Means for automatically driving a vehicle or assisting a driver" refers to means including a function for automatically controlling the operation of a vehicle based on the generated parking trajectory, or a function for providing guidelines or voice guidance to support the driver when driving by himself.

[0910] "Means for executing artificial intelligence algorithms" refers to means for using AI technology to analyze images of the surrounding area of ​​the parking lot and identify the location of vacant spaces and obstacles.

[0911] "Means for generating parking guidelines and displaying them on the screen" refers to a means for generating guidelines for the driver to follow based on the recognized parking lot information and displaying them on a smartphone or in-car display.

[0912] The present invention is a system for assisting drivers in parking and leaving narrow parking spaces safely and efficiently. The system includes the following main means:

[0913] 1. A means of capturing images of the surroundings

[0914] The server collects image data using cameras and sensors to understand the surrounding environment of the parking lot. These cameras are installed at the entrance to the parking lot and around each parking space. The image data is acquired in real time and sent to the system.

[0915] 2. How to obtain parking rules

[0916] The server retrieves information about the parking lot, such as usage rules, layout, available spaces, etc. from a database. This information is regularly updated by the parking lot manager and provided to the server.

[0917] 3. A means of inputting vehicle size and specifications

[0918] Users can input detailed information such as vehicle length, width, height, and axle distance through the terminal, which is used to calculate the optimal parking trajectory.

[0919] 4. Means for generating optimal parking trajectories

[0920] The server uses an AI algorithm to calculate and generate a safe and efficient parking route based on the acquired image data and vehicle information. This algorithm determines the optimal trajectory by taking into account factors such as approach angle, steering operation, and timing of backing up.

[0921] 5. Means for autonomous driving or driver assistance of vehicles

[0922] The device provides a function to automatically control vehicle operation based on the generated parking trajectory. It also supports the driver by providing parking guidelines and voice guidance when driving independently. This function displays parking guidelines on a smartphone or in-car display to help the driver park according to them.

[0923] 6. Means for running artificial intelligence algorithms

[0924] The server uses AI technology to analyze images of the surrounding parking lot and identify the location of available spaces and obstacles. This AI algorithm uses a deep learning model to suggest the optimal parking strategy based on the results of the image analysis.

[0925] 7. A method for generating parking guidelines and displaying them on the screen

[0926] The device generates guidelines for the driver to follow based on the recognized parking information, displays them on a smartphone or in-car display, and uses voice synthesis to provide driving instructions at the appropriate time.

[0927] Specific examples

[0928] For example, a user arrives at a small parking lot in central Tokyo and launches a parking support app on their smartphone. The app scans the entire parking lot using a camera and calculates the optimal parking route using an AI algorithm. The user parks their car according to the guidelines displayed on the smartphone screen and uses voice guidance to perform the appropriate operations, completing the parking process with peace of mind. An example of a prompt that can be applied at this time is, "Please recognize the parking lot layout and identify the location of available spaces and obstacles."

[0929] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0930] Step 1:

[0931] Input: Real-time image data from a parking lot camera.

[0932] Processing: The server uses the parking lot cameras to capture images of the surrounding area, including footage of the parking space and any obstacles in the area.

[0933] Output: Captured parking lot image data.

[0934] Step 2:

[0935] Input: Parking rules, available space information, and image data.

[0936] Processing: The server collects information about parking rules and available spaces from a database and compares it with the captured image data, for example, identifying no-parking areas and the location of obstacles.

[0937] Output: Parking lot layout information as a result of image processing (available parking areas, empty spaces, obstacle locations).

[0938] Step 3:

[0939] Input: Vehicle size and specifications (length, width, height, axle distance).

[0940] Processing: The user uses a device (smartphone) to input vehicle information, which is then sent to the server.

[0941] Output: Transmitted vehicle information data.

[0942] Step 4:

[0943] Input: Parking lot layout information, vehicle information.

[0944] Processing: The server uses AI algorithms to generate the optimal parking trajectory based on the input vehicle information and parking lot layout information, for example, calculating the parking space entry angle and backing timing.

[0945] Output: Generated parking trajectory data.

[0946] Step 5:

[0947] Input: Parking trajectory data.

[0948] Processing: Based on the generated optimal parking trajectory, the device generates parking guidelines to assist the driver and displays them on a smartphone or in-car display. It also uses the voice guidance function to provide driving instructions at the appropriate time.

[0949] Output: Parking guideline display data and voice guidance data provided to the user.

[0950] Step 6:

[0951] Input: User action to prepare for delivery.

[0952] Processing: When the user begins preparing to leave, the server again captures images of the surroundings and uses AI algorithms to generate an optimal exit trajectory, taking into account any new obstacles or changing parking lot conditions.

[0953] Output: Generated outbound trajectory data.

[0954] Step 7:

[0955] Input: Outbound trajectory data.

[0956] Processing: Based on the generated departure trajectory, the terminal generates departure guidelines to assist the driver and displays them on a smartphone or in-car display. It also uses a voice guidance function to provide departure instructions at the appropriate time.

[0957] Output: Display data and voice guidance data for delivery guidelines provided to the user.

[0958] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0959] The present invention is a system that assists drivers in parking in tight spaces safely and efficiently, and further combines it with an emotion engine that recognizes the user's emotions. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automatic driving or driver assistance, as well as analyzing the user's emotions.

[0960] Explanation of program processing

[0961] 1. Acquiring surrounding images and understanding parking lot rules

[0962] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[0963] 2. Enter your car information

[0964] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[0965] 3. Parking Trajectory Generation

[0966] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[0967] 4. Implementation or assistance of automated driving

[0968] In Autonomous mode:

[0969] The device will then perform autonomous driving based on the generated optimal trajectory. In this mode, the device will automatically control steering, acceleration, and deceleration to complete the parking operation.

[0970] In driver assistance mode:

[0971] The device displays the generated optimal trajectory on an in-car monitor and assists the user in driving by following the displayed guidelines. The device also prompts the user to take action at the appropriate time using audio and visual alerts.

[0972] 5. User Emotion Recognition

[0973] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling stressed or anxious, that emotion is conveyed to the system.

[0974] 6. Emotion-based system adjustment

[0975] The server adjusts the system's behavior based on the recognized user's emotions.

[0976] If the user is feeling stressed, the device will play relaxing music and gently guide them through operations.

[0977] If the user feels anxious or nervous, the device will preferentially select the driving assistance mode and provide detailed guidance on the parking trajectory.

[0978] 7. New trajectory generation when leaving the warehouse

[0979] After parking, when the user starts preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. Based on the input information, it generates the optimal trajectory for leaving the parking lot.

[0980] 8. Conducting or assisting with deliveries

[0981] In Autonomous mode:

[0982] The terminal automatically drives the vehicle according to the generated departure trajectory and causes the vehicle to leave the parking lot.

[0983] In driver assistance mode:

[0984] The terminal displays the exit trajectory on the in-car monitor and guides the user through the exit process by following the displayed guidelines, using audio and visual alerts to encourage appropriate operation.

[0985] Specific examples

[0986] Specific example of parking

[0987] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly. The device also recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[0988] Specific example of shipping

[0989] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates a new optimal trajectory for leaving. The device then either leaves the vehicle in autonomous driving mode or displays the trajectory in driver assistance mode and helps the user leave the vehicle accordingly. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[0990] The processing flow will be explained below.

[0991] Step 1:

[0992] The server acquires current surrounding images from a camera installed at the entrance of the parking lot, which are used to recognize the layout of parking spaces and the location of obstacles.

[0993] Step 2:

[0994] The server collects information on parking rules and available spaces from the database, which allows users to know the exact location of available spaces and important points to note when parking.

[0995] Step 3:

[0996] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[0997] Step 4:

[0998] The device sends the entered vehicle information to the server, which then receives all the data necessary for analysis.

[0999] Step 5:

[1000] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[1001] Step 6:

[1002] The server then transmits the generated optimal parking trajectory data to the terminal, allowing the terminal to learn the specific parking method.

[1003] Step 7:

[1004] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, it uses facial expression analysis and voice analysis technology to detect whether the user is feeling stressed or anxious.

[1005] Step 8:

[1006] When the emotion engine recognizes emotions such as stress or anxiety, the device automatically adjusts system behavior, including playing relaxing music and providing more detailed operating instructions.

[1007] Step 9:

[1008] When parking in autonomous driving mode based on the received parking trajectory, the device automatically performs steering, acceleration, and deceleration, thereby completing parking safely and efficiently.

[1009] Step 10:

[1010] When the device is in driver assistance mode, the generated parking trajectory is displayed on the in-car monitor. The user drives and parks according to this display. The device prompts the user to operate the vehicle at the appropriate time using audio and visual alerts.

[1011] Step 11:

[1012] After parking is complete, the terminal notifies the user, allowing the user to confirm that parking was completed successfully.

[1013] Step 12:

[1014] To start preparation for delivery, the user issues an instruction to the system, which makes the server ready to acquire new information.

[1015] Step 13:

[1016] The server then acquires images of the surrounding area again to obtain the latest information, thereby ensuring safety when the vehicle leaves the warehouse.

[1017] Step 14:

[1018] The server generates an optimal trajectory for the departure, calculating a new departure route and taking into account the location of obstacles and the movement of other vehicles.

[1019] Step 15:

[1020] The server then transmits the generated retrieval trajectory data to the terminal, which then retrieves the vehicle based on this information.

[1021] Step 16:

[1022] The terminal either drives automatically along the departure trajectory or displays the information on a monitor to assist the user in leaving the parking lot. In the autonomous driving mode, the terminal performs all operations automatically. In the driver assistance mode, the user operates the device themselves.

[1023] Step 17:

[1024] After the vehicle has left the parking lot, the terminal notifies the user, allowing the user to confirm that the vehicle has safely left the parking lot.

[1025] Example 2

[1026] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1027] Parking safely and efficiently in tight parking spaces is a challenging task for many drivers. The stress and anxiety drivers feel during parking also impact safety. However, current automated driving and driver assistance systems lack the ability to recognize user emotions and adapt to the user's psychological state during the parking process. Therefore, there is a need for a parking assistance system that can adjust its behavior based on the user's emotions.

[1028] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1029] In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting the size and specifications of the vehicle, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for recognizing the user's emotions, and means for adjusting the system operation based on the recognized user's emotions, thereby making it possible to provide optimal parking support while taking the user's emotions into consideration.

[1030] "Means for acquiring images of the surrounding area" refers to technology that collects real-time images of the surrounding area using cameras and sensors installed at the entrance to the parking lot.

[1031] "Means for obtaining parking lot rules" refers to technology for obtaining traffic rules and available space information within a parking lot from a database.

[1032] "Means for inputting vehicle dimensions and specifications" refers to technology that provides an interface for users to input the dimensions and specifications (e.g., length, width, height, axle distance, etc.) of their vehicle.

[1033] "Means for generating optimal parking trajectories" refers to technology that uses AI algorithms to calculate entry angles, steering operation, timing of backing up, etc. based on acquired surrounding images and vehicle information to generate optimal parking trajectories.

[1034] "Means for autonomously driving a vehicle or assisting the driver" refers to technology that automatically controls the vehicle's steering, acceleration, and deceleration based on the generated parking trajectory, or technology that displays the trajectory on an in-vehicle monitor and assists the driver with audio and visual alerts.

[1035] "Means for recognizing user emotions" refers to an emotion engine that analyzes emotions such as stress and anxiety from the user's facial expressions and voice.

[1036] "Means for adjusting system behavior" refers to functions that change the behavior of the system based on the recognized user's emotions, such as playing relaxing music or providing gentler operating guidance.

[1037] This invention combines a system that assists drivers in parking in tight spaces safely and efficiently with an emotion engine that recognizes the user's emotions. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automatic driving or driver assistance, as well as analyzing the user's emotions.

[1038] The detailed system configuration uses the following hardware and software:

[1039] Acquiring surrounding images and understanding parking rules

[1040] The server obtains current surrounding images from a camera installed at the entrance to the parking lot. This high-resolution camera can clearly recognize obstacles and available spaces within the parking lot. Image analysis is performed in real time using image recognition software such as OpenCV. At the same time, the server collects information on parking lot rules and available spaces from a database. This database records traffic rules within the parking lot and space usage status.

[1041] Enter your car information

[1042] The device provides an interface for the user to input vehicle dimensions and specifications (e.g., length, width, height, and axle distance). This interface may take the form of a smartphone app or an in-vehicle display. The input information is stored on the device and sent to a server to avoid having to enter it again.

[1043] Parking trajectory generation

[1044] The server performs data analysis based on the acquired surrounding images and vehicle information. AI algorithms (e.g., TensorFlow and PyTorch) are used for the analysis, calculating factors such as approach angle, steering operation, and timing of backing up, and generating the optimal parking trajectory. The generated trajectory information is sent to the device.

[1045] Implementation or assistance of automated driving

[1046] In autonomous driving mode, the device automatically drives the vehicle based on the generated optimal trajectory. In this mode, the device automatically controls steering, acceleration, and deceleration, and completes parking using dedicated autonomous driving software (e.g., Autoware).

[1047] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. The device uses audio and visual alerts to prompt the user to take action at the appropriate time.

[1048] User Emotion Recognition

[1049] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. Emotion analysis utilizes facial and voice analysis software such as Microsoft Azure Face API and Google Speech-to-Text. For example, if the user is feeling stressed or anxious, that emotion is conveyed to the system.

[1050] Emotion-based system adjustment

[1051] The server adjusts the system's behavior based on the user's recognized emotions. Specifically, if the user feels stressed, the device will play relaxing music (e.g., using Spotify or Apple Music APIs) and provide gentle guidance on operations. If the user feels anxious or tense, the device will prioritize driving assistance mode and provide detailed guidelines.

[1052] New trajectory generation when leaving the warehouse

[1053] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area to ascertain the latest situation. Based on the input information, an AI algorithm is used to generate the optimal trajectory for leaving the parking lot and send it to the terminal.

[1054] Conducting or assisting with shipping

[1055] In autonomous driving mode, the device automatically drives the vehicle according to the generated exit trajectory and leaves the parking lot. In driver assistance mode, the device displays the exit trajectory on the in-car monitor and helps the user leave the parking lot by following the displayed guidelines. Audio and visual alerts are also used during this process.

[1056] Specific examples

[1057] As a concrete example of parking, when a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device that receives this information either parks in autonomous driving mode or displays the trajectory on a monitor in driver assistance mode, helping the user park accordingly. At the same time, the device recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[1058] Example prompts to input to the generative AI model:

[1059] "A user arrives at a parking lot, and the system automatically finds a parking space and assists the user. If the user is emotionally stressed, how will the system respond?"

[1060] For example, when the user prepares to leave the parking lot after parking, the server acquires images of the surrounding area again and generates a new optimal trajectory for leaving. The device then either leaves the parking lot in autonomous driving mode or displays the trajectory in driver assistance mode and assists the user in leaving the parking lot. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[1061] Example prompts to input to the generative AI model:

[1062] "When the user attempts to leave the parking lot after parking, please explain how the system will assist the user in leaving the parking lot and how it will respond depending on the user's emotions."

[1063] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1064] Step 1: Acquire surrounding images and understand parking rules

[1065] The server obtains real-time images of the surrounding area from a camera installed at the entrance to the parking lot. The camera image is used as input, and the obtained image data is obtained as output. This image data is analyzed to recognize the layout of parking spaces and the location of obstacles. Image recognition software such as OpenCV is used for image analysis. At the same time, the server obtains information on parking lot rules and available spaces from a database. The information registered by the user at the parking lot is used as input, and the obtained parking lot rule data is obtained as output.

[1066] Specific behavior:

[1067] The server receives the video stream from the camera.

[1068] The server performs image analysis and recognizes parking spaces and obstacles.

[1069] The server retrieves the parking rules from the database.

[1070] Step 2: Enter your vehicle information

[1071] The terminal allows the user to input the vehicle size and specifications (length, width, height, axle distance, etc.). The user's input data is used as input, and stored vehicle information is obtained as output. This information is saved on the terminal and later sent to the server. The stored data is reused to avoid having to enter the information again.

[1072] Specific behavior:

[1073] The device displays the vehicle information input screen.

[1074] The user inputs the vehicle size and specifications.

[1075] The device saves the input data and sends it to the server.

[1076] Step 3: Generate parking trajectories

[1077] The server performs data analysis based on the acquired surrounding images and the user's vehicle information. Surrounding image data and vehicle information data are used as input, and parking trajectory data is generated as output. Here, AI algorithms (e.g., TensorFlow and PyTorch) are used to calculate factors such as approach angle, steering operation, and timing of backing up, and generate the optimal parking trajectory.

[1078] Specific behavior:

[1079] The server acquires surrounding images and vehicle information.

[1080] The server analyzes using an AI algorithm.

[1081] The server generates the optimal parking trajectory and sends it to the device.

[1082] Step 4: Implementing or Assisting Autonomous Driving

[1083] In autonomous driving mode, the device performs autonomous driving based on the generated optimal trajectory. Parking trajectory data is used as input, and the parking completion status is obtained as output. Dedicated autonomous driving software (e.g., Autoware) is used.

[1084] Specific behavior:

[1085] The terminal receives the optimal orbit data.

[1086] The device begins autonomous driving using Autoware.

[1087] The device controls the steering wheel, accelerator, and brakes.

[1088] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user to drive according to the guideline. The input is the parking trajectory data, and the output is driving assistance for the user. The device uses audio and visual alerts.

[1089] Specific behavior:

[1090] The terminal displays the optimal orbit data on the monitor.

[1091] The user drives according to the guidelines.

[1092] The device assists the user with voice alerts.

[1093] Step 5: Recognizing User Emotions

[1094] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. The user's facial expression and voice data are used as input, and the recognized emotion data is obtained as output. Emotion analysis uses Microsoft Azure Face API and Google Speech-to-Text, among others.

[1095] Specific behavior:

[1096] The device uses a camera and microphone to collect the user's facial expressions and voice.

[1097] The device analyzes using an emotion engine.

[1098] The device sends the analysis results to the server.

[1099] Step 6: Adjusting the system based on emotions

[1100] The server adjusts the system's behavior based on the recognized user emotion. Emotional data is used as input and the adjusted system behavior is obtained as output. If the user is feeling stressed, the device will play relaxing music (using Spotify or Apple Music APIs) or provide gentle guidance.

[1101] Specific behavior:

[1102] The server processes the emotion data.

[1103] The device will select and play the appropriate music.

[1104] The device provides a friendly audio guide.

[1105] Step 7: New trajectory generation for exit

[1106] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. The surrounding image data is used as input, and the optimal trajectory data for leaving is obtained as output. The optimal trajectory for leaving is generated using an AI algorithm and sent to the terminal.

[1107] Specific behavior:

[1108] The user begins preparation for delivery.

[1109] The server acquires surrounding images from the camera.

[1110] The server generates the optimal trajectory for delivery and sends it to the terminal.

[1111] Step 8: Perform or assist with shipping

[1112] In the automatic driving mode, the terminal automatically drives according to the generated retrieval trajectory. The retrieval trajectory data is used as input, and the retrieval completion status is obtained as output.

[1113] Specific behavior:

[1114] The terminal receives the delivery trajectory data.

[1115] The terminal will execute the retrieval in automatic operation mode.

[1116] In driver assistance mode, the device displays the exit trajectory on the in-vehicle monitor and assists the user in following the guidelines when leaving the vehicle. The exit trajectory data is used as input, and driving assistance is provided to the user as output. Audio and visual alerts encourage appropriate operation.

[1117] Specific behavior:

[1118] The terminal displays the delivery trajectory data on the monitor.

[1119] The user issues goods according to the guidelines.

[1120] The device will prompt you with an audio alert.

[1121] (Application example 2)

[1122] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1123] Modern parking lots often have narrow spaces and complex layouts, making parking difficult for drivers. The stress and anxiety drivers feel when parking can negatively impact driving safety and efficiency. Even in these situations, there is a demand for systems that support safe and efficient parking. Furthermore, systems must provide support not only when parking but also when exiting the parking lot safely and efficiently.

[1124] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting vehicle size and specifications, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for analyzing user emotions, and means for adjusting system operation based on the user emotions. This enables the driver to park safely and efficiently, and by providing support according to the driver's emotions, it becomes possible to reduce stress and anxiety.

[1125] The "surrounding image" is an image acquired using a device such as a camera to recognize the layout of the parking lot and the location of obstacles.

[1126] "Parking rules" are rules and instructions regarding parking spaces, prohibited areas, and how to pass through a parking lot.

[1127] "Vehicle size and specifications" refers to information that refers to the physical dimensions and characteristics of a vehicle, such as length, width, height, and axle distance.

[1128] A "parking trajectory" is a planned path for a vehicle to park in a parking space, including the optimal approach angle, steering operation, and timing of backing up.

[1129] "Autonomous driving" is the ability of a vehicle to perform driving operations autonomously without human intervention.

[1130] "Supporting the driver" means providing assistance by providing guidelines, alerts, etc. when the driver parks or leaves the vehicle by themselves.

[1131] "User's emotions" refer to psychological states such as stress, anxiety, joy, etc. that are recognized from the user's facial expressions, voice, etc.

[1132] "Analyzing emotions" refers to the process of analyzing the user's facial expressions and voice to identify their emotions.

[1133] "Adjusting the system's behavior" means changing the system's functions and behavior in response to the user's emotions, for example, by playing relaxing music or providing gentle guidance.

[1134] This invention relates to a system that assists drivers in parking in narrow parking spaces safely and efficiently. The system includes multiple means for capturing images of the surrounding area, understanding parking rules, inputting vehicle information, generating an optimal parking trajectory, and performing automated driving or driver assistance and user emotion analysis.

[1135] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[1136] The terminal prompts the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.). This information is stored in the terminal, eliminating the need to input it again. The server generates the optimal parking trajectory based on the captured surrounding images and vehicle information. The server uses an AI algorithm (e.g., TensorFlow) to calculate factors such as approach angle, steering operation, and timing for backing up, and generates the optimal parking trajectory.

[1137] In autonomous driving mode, the device drives autonomously based on the generated optimal trajectory. In this mode, the device automatically controls steering, acceleration, and deceleration to complete parking. In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. The device uses audio and visual alerts to prompt the user to operate at the appropriate time.

[1138] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling stressed or anxious, that emotion is communicated to the system. The server adjusts the system's operation based on the recognized user emotion. If the user is feeling stressed, the device may play relaxing music or gently guide the user through the operation. If the user is feeling anxious or tense, the device may prioritize driving assistance mode and provide detailed guidance on the parking trajectory.

[1139] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area and ascertains the latest situation. Based on the input information, it generates an optimal trajectory for leaving. In autonomous driving mode, the device automatically drives the vehicle according to the generated trajectory and leaves the parking lot. In driver assistance mode, the device displays the trajectory on the in-car monitor and assists the user in leaving the parking lot by following the displayed guidelines. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[1140] Specific examples

[1141] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly. The device also recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[1142] Prompt Sentence Examples

[1143] "A user is trying to park a car in a tight parking space. Implement a system that recognizes whether the user is stressed or not from their facial expression, and generates an optimal parking trajectory to assist the user in parking in real time."

[1144] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1145] Step 1:

[1146] The server obtains current surrounding images from a camera installed at the entrance to the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. The video data obtained from the camera (input) is used to identify the location of vacant spaces in the parking lot and obstacles (example of data processing: object detection using image processing algorithms). The output becomes map information of the parking lot.

[1147] Step 2:

[1148] The server collects information about parking rules and available spaces from a database. This information (input) includes rules and instructions about permitted parking spaces, prohibited areas, and how to navigate. The output is the optimal parking location that complies with the permitted parking areas and rules. This is achieved by executing a database query to obtain the required information.

[1149] Step 3:

[1150] The terminal allows the user to input the vehicle size and specifications (e.g. length, width, height, axle distance, etc.). This information (input) is stored in a vehicle database. The (output) is the complete vehicle dimensions, held in a reusable format. Input is received through a user interface.

[1151] Step 4:

[1152] The server generates an optimal parking trajectory based on the acquired surrounding images and vehicle information. It uses an AI algorithm (e.g., TensorFlow) to calculate the approach angle, steering operation, timing of backing up, etc. (example of data calculation: application of a trajectory prediction model using a neural network) and generates the generated parking trajectory (output).

[1153] Step 5:

[1154] In autonomous driving mode, the device performs autonomous driving based on the generated optimal trajectory. It automatically controls steering, acceleration, and deceleration (vehicle operation using a control algorithm) and completes parking (output). The input is optimal trajectory information. Sensor information is fed back in real time to control the vehicle's movement.

[1155] Step 6:

[1156] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. It uses audio and visual alerts to prompt the user to operate the vehicle at the appropriate time (input is the generated parking trajectory information, output is guideline display and alerts) to complete parking. It generates audio and visual guides and displays them on the user interface.

[1157] Step 7:

[1158] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. It analyzes the user's facial images and voice data (input) (example of data processing: facial expression and voice analysis using machine learning models) and identifies the user's state of stress or anxiety (output). It then identifies the user's emotional state based on the results of the data analysis.

[1159] Step 8:

[1160] The server adjusts the system's behavior based on the recognized user emotion. If the user feels stressed, the device plays relaxing music (input: emotion recognition result, output: music playback) and gently guides the user through the operation. If the user feels anxious or tense, the device prioritizes driving assistance mode and provides more detailed guidance (output), adjusts system settings, and provides appropriate feedback.

[1161] Step 9:

[1162] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. The acquired image data (input) is used to re-recognize the surrounding environment (example of data processing: re-application of object detection algorithm) and generate the optimal trajectory for leaving (output). The optimal exit route is calculated through data analysis.

[1163] Step 10:

[1164] In autonomous driving mode, the terminal automatically drives the vehicle according to the generated departure trajectory and leaves the parking lot. It performs autonomous driving control based on the departure trajectory information (input) and optimizes the vehicle's movement (output). It executes departure operations including real-time feedback control.

[1165] Step 11:

[1166] In driver assistance mode, the device displays the exit trajectory on the in-car monitor and assists the user in following the displayed guidelines to complete the exit. Audio and visual alerts are used to prompt the user to take the appropriate action and complete the exit (input is the exit trajectory information, output is the guideline display and alerts). The device makes full use of the user interface to provide appropriate guidance and feedback.

[1167] Example prompt: "A user is trying to park a car in a narrow parking space. Implement a system that recognizes whether the user is stressed from their facial expression and generates an optimal parking trajectory to assist the user in parking in real time."

[1168] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1169] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1170] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1171] [Fourth embodiment]

[1172] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1173] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1174] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1175] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1176] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1177] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1178] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1179] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1180] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1181] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1182] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1183] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1184] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1185] The present invention is a system that assists drivers in parking in tight spaces safely and efficiently. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automated driving or driver assistance.

[1186] Explanation of program processing

[1187] 1. Acquiring surrounding images and understanding parking lot rules

[1188] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[1189] 2. Enter your car information

[1190] The terminal provides a means for the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) so that the system can obtain accurate vehicle information to use in calculating the optimal parking trajectory.

[1191] 3. Parking Trajectory Generation

[1192] The server uses the captured images of the surroundings and vehicle information to generate the optimal parking trajectory, using an AI algorithm to calculate factors such as approach angle, steering operation, and timing for backing up, providing a route for safely and reliably entering a parking space.

[1193] 4. Implementation or assistance of automated driving

[1194] In Autonomous mode:

[1195] The device will then perform autonomous driving based on the generated optimal trajectory. In this mode, the device will automatically control steering, acceleration, and deceleration to complete the parking operation.

[1196] In driver assistance mode:

[1197] The device displays the generated optimal trajectory on an in-car monitor and assists the user in driving by following the displayed guidelines. The device also uses audio and visual alerts to support the user in taking appropriate actions at the appropriate time.

[1198] 5. New trajectory generation when leaving the warehouse

[1199] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates an optimal trajectory for leaving.

[1200] In Autonomous mode:

[1201] The terminal automatically drives the vehicle according to the generated departure trajectory and causes the vehicle to leave the parking lot.

[1202] In driver assistance mode:

[1203] The terminal displays the exit trajectory on the in-car monitor and assists the user in following the displayed guidelines. During exit, audio and visual alerts are also used to encourage appropriate operation.

[1204] Specific examples

[1205] Specific example of parking

[1206] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks the car in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly.

[1207] Specific example of shipping

[1208] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates a new optimal trajectory for leaving. The device then leaves the parking lot in autonomous driving mode or displays the trajectory in driver assistance mode, helping the user to follow it. This process allows the user to park and leave the parking lot with peace of mind.

[1209] The processing flow will be explained below.

[1210] Step 1:

[1211] The server acquires current surrounding images from a camera installed at the entrance of the parking lot, which are used to recognize the layout of parking spaces and the location of obstacles.

[1212] Step 2:

[1213] The server collects information on parking rules and available spaces from the database, which allows users to know the exact location of available spaces and important points to note when parking.

[1214] Step 3:

[1215] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[1216] Step 4:

[1217] The device sends the entered vehicle information to the server, which then receives all the data necessary for analysis.

[1218] Step 5:

[1219] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[1220] Step 6:

[1221] The server then transmits the generated optimal parking trajectory data to the terminal, allowing the terminal to learn the specific parking method.

[1222] Step 7:

[1223] When parking in autonomous driving mode based on the received parking trajectory, the device automatically performs steering, acceleration, and deceleration, thereby completing parking safely and efficiently.

[1224] Step 8:

[1225] When the device is in driver assistance mode, the generated parking trajectory is displayed on the in-car monitor. The user drives and parks according to this display. The device prompts the user to operate the vehicle at the appropriate time using audio and visual alerts.

[1226] Step 9:

[1227] After parking is complete, the terminal notifies the user, allowing the user to confirm that parking was completed successfully.

[1228] Step 10:

[1229] To start preparation for delivery, the user issues an instruction to the system, which makes the server ready to acquire new information.

[1230] Step 11:

[1231] The server then acquires images of the surrounding area again to obtain the latest information, thereby ensuring safety when the vehicle leaves the warehouse.

[1232] Step 12:

[1233] The server generates an optimal trajectory for the departure, calculating a new departure route and taking into account the location of obstacles and the movement of other vehicles.

[1234] Step 13:

[1235] The server then transmits the generated retrieval trajectory data to the terminal, which then retrieves the vehicle based on this information.

[1236] Step 14:

[1237] The terminal either drives automatically along the departure trajectory or displays the information on a monitor to assist the user in leaving the parking lot. In the autonomous driving mode, the terminal performs all operations automatically. In the driver assistance mode, the user operates the device themselves.

[1238] Step 15:

[1239] After the vehicle has left the parking lot, the terminal notifies the user, allowing the user to confirm that the vehicle has safely left the parking lot.

[1240] Example 1

[1241] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1242] Current parking assistance systems lack the precision and convenience required to achieve safe and efficient parking in tight parking spaces. The ability to quickly respond to changes in the surrounding environment is particularly important, and safety when exiting the parking space is equally important. Furthermore, the provision of interfaces and guidance to reduce the driver's burden is insufficient. The present invention aims to solve these problems.

[1243] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1244] In this invention, the server includes a means for acquiring surrounding conditions, a means for acquiring parking lot rule information, a means for inputting detailed vehicle information, a means for generating an optimal parking route based on the acquired images and vehicle information, and a means for automatically operating the vehicle or assisting the driver based on the generated parking route, thereby enabling the user to safely and efficiently park and exit even in a narrow parking space.

[1245] "Surrounding conditions" refers to information that indicates the state of objects and space around the vehicle.

[1246] "Parking lot rule information" refers to data regarding parking rules and available spaces that apply within a parking lot.

[1247] "Vehicle details" refers to specific data about the vehicle's dimensions and specifications (e.g., length, width, height, distance between axles, etc.).

[1248] An "optimal parking route" is the best calculated path for a vehicle to safely and efficiently enter a parking space.

[1249] "Autonomous operation" means that the system automatically controls the operation of the vehicle, including steering, acceleration, and deceleration.

[1250] "Means to assist the driver" refers to the system's ability to provide appropriate driving assistance to the driver in the form of voice or visual alerts, guidelines, etc.

[1251] An "optimal route for exit" is the calculated best path for a vehicle to safely exit a parking space.

[1252] An "in-vehicle display device" refers to a monitor or display device installed inside a vehicle that displays information to the driver.

[1253] The present invention is a system that assists drivers in parking and leaving narrow parking spaces safely and efficiently. The system includes a series of means for acquiring the surrounding situation, understanding parking lot rule information, inputting detailed vehicle information, generating optimal parking and leaving routes, and automatically operating the vehicle or assisting the driver based on the generated routes.

[1254] First, the server acquires the surrounding situation using a high-resolution camera installed at the entrance to the parking lot. This camera captures the surrounding image in real time and recognizes the layout of parking spaces and the location of obstacles using an object detection algorithm based on deep learning. The server also obtains information on parking lot rules and available spaces from a database. This enables accurate calculation of the parking route.

[1255] Next, the device provides a means for the user to input detailed vehicle information (e.g., length, width, height, axle distance, etc.). The input information is stored in a database and made available to the server. Based on this information, the server generates an optimal parking route. Specifically, it uses AI algorithms to calculate entry angles, steering operations, timing of backing, etc., and provides a route for safely and reliably entering a parking space. Specific technologies used include deep learning and reinforcement learning.

[1256] Based on the generated parking route, the device parks in either autonomous mode or driver-assisted mode. In autonomous mode, the device automatically controls steering, acceleration, and deceleration to complete the parking. In driver-assisted mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. Audio and visual alerts are used to help the user perform operations at the appropriate time.

[1257] After parking, when the user begins preparing to leave, the server again acquires information about the surrounding area and generates the optimal route for leaving. Both autonomous driving mode and driver assistance mode are available during leaving, allowing the user to leave the parking lot with peace of mind.

[1258] For example, you can instruct a generative AI model using specific prompts like the following:

[1259] 1. "Describe an automated driving assistance system for tight parking spaces."

[1260] 2. "How can I calculate the optimal parking route using vehicle and parking information?"

[1261] 3. "Please explain the difference between autonomous and driver assistance modes."

[1262] These prompts allow the system to properly obtain the necessary information and provide optimal parking and exit assistance.

[1263] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1264] Step 1: Acquire an image of the surroundings

[1265] The server acquires images of the surrounding area using a high-resolution camera installed at the entrance to the parking lot. This camera captures images in real time, and the server receives the image data. The acquired image data is analyzed using a deep learning algorithm to recognize the layout of parking spaces and the location of obstacles. The input is the camera's image data, and the output is the analyzed image data.

[1266] Step 2: Know the parking rules

[1267] The server retrieves rules information from a parking lot database. This database contains parking rules and available space information. For example, it checks whether a particular parking space is reserved or whether an obstacle exists. The input is a request for parking rules information, and the output is the parking rules and available space information.

[1268] Step 3: Enter vehicle information

[1269] The terminal provides a means for the user to input vehicle details (length, width, height, axle distance, etc.). The user enters these details through an input form. The input is the vehicle information entered by the user, and the output is the vehicle information stored in a database.

[1270] Step 4: Generate optimal parking trajectory

[1271] The server generates the optimal parking trajectory based on the surrounding image obtained in step 1, the parking lot rules acquired in step 2, and the detailed vehicle information entered in step 3. It uses AI algorithms (including deep learning and reinforcement learning) to calculate the approach angle, steering operation, timing of backing up, etc. The inputs are image data, parking lot rules, and vehicle information, and the output is the optimal parking route.

[1272] Step 5: Implementing or Assisting Autonomous Driving

[1273] In Autonomous mode:

[1274] The terminal performs autonomous driving based on the optimal parking trajectory received from the server. It controls the engine, steering, brakes, etc., and automatically parks the vehicle according to the instructed trajectory. The input is the optimal parking route, and the output is the completion of automatic parking.

[1275] In driver assistance mode:

[1276] The terminal displays the generated optimal trajectory on the monitor inside the vehicle and assists the user in parking by following the displayed guidelines. Guidelines for approach angles and steering operations are displayed on the navigation screen, and audio and visual alerts are used to prompt the user to perform operations at the appropriate time. The input is the optimal parking route, and the output is the completion of driver assistance.

[1277] Step 6: New trajectory generation for exit

[1278] When the user starts preparing to leave, the server acquires images of the surroundings again and generates an optimal trajectory for leaving. It calculates a safe route to leave, taking into account the position information of other vehicles and obstacles in the parking lot. The input is the latest surrounding images and information about the parking lot, and the output is the optimal route to leave.

[1279] Step 7: Conduct or assist with shipping

[1280] In Autonomous mode:

[1281] The terminal automatically drives the vehicle based on the generated departure trajectory and causes it to leave the parking lot. The input is the optimal departure route, and the output is the completion of automatic departure.

[1282] In driver assistance mode:

[1283] The terminal displays the departure trajectory on the in-vehicle monitor and assists the user in following the displayed guidelines. Audio and visual alerts are used to prompt appropriate operation. The input is the optimal departure route, and the output is completion of the driver's assistance with leaving the vehicle.

[1284] (Application example 1)

[1285] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1286] It is often difficult for drivers to safely and efficiently park in narrow parking spaces, especially in urban areas and crowded parking lots. Current parking assistance systems are insufficient, and further support is needed to enable drivers to park and exit safely. Another issue is the lack of technology that can identify available parking spaces and the location of obstacles in a parking lot in real time and provide the optimal parking method.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1288] In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting the size and specifications of the vehicle, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for executing an artificial intelligence algorithm for recognizing the acquired images of the surroundings of the parking lot, and means for generating parking guidelines based on the recognized position information of vacant spaces and obstacles in the parking lot and displaying them on a screen. This enables the driver to park and exit a narrow parking space safely and efficiently with peace of mind.

[1289] The "means for acquiring images of the surroundings" refers to a means for collecting image data using a camera or a sensor in order to grasp the surrounding environment of the vehicle.

[1290] "Means for obtaining parking lot rules" refers to a means for obtaining information such as parking lot usage rules, layout, and available spaces from a database.

[1291] "Means for inputting vehicle size and specifications" refers to a means for a user to input detailed information such as the length, width, height, and axle distance of a vehicle.

[1292] The "means for generating the optimal parking trajectory" is a means for calculating and generating a safe and efficient parking route using an AI algorithm based on acquired image data and vehicle information.

[1293] "Means for automatically driving a vehicle or assisting a driver" refers to means including a function for automatically controlling the operation of a vehicle based on the generated parking trajectory, or a function for providing guidelines or voice guidance to support the driver when driving by himself.

[1294] "Means for executing artificial intelligence algorithms" refers to means for using AI technology to analyze images of the surrounding area of ​​the parking lot and identify the location of vacant spaces and obstacles.

[1295] "Means for generating parking guidelines and displaying them on the screen" refers to a means for generating guidelines for the driver to follow based on the recognized parking lot information and displaying them on a smartphone or in-car display.

[1296] The present invention is a system for assisting drivers in parking and leaving narrow parking spaces safely and efficiently. The system includes the following main means:

[1297] 1. A means of capturing images of the surroundings

[1298] The server collects image data using cameras and sensors to understand the surrounding environment of the parking lot. These cameras are installed at the entrance to the parking lot and around each parking space. The image data is acquired in real time and sent to the system.

[1299] 2. How to obtain parking rules

[1300] The server retrieves information about the parking lot, such as usage rules, layout, available spaces, etc. from a database. This information is regularly updated by the parking lot manager and provided to the server.

[1301] 3. A means of inputting vehicle size and specifications

[1302] Users can input detailed information such as vehicle length, width, height, and axle distance through the terminal, which is used to calculate the optimal parking trajectory.

[1303] 4. Means for generating optimal parking trajectories

[1304] The server uses an AI algorithm to calculate and generate a safe and efficient parking route based on the acquired image data and vehicle information. This algorithm determines the optimal trajectory by taking into account factors such as approach angle, steering operation, and timing of backing up.

[1305] 5. Means for autonomous driving or driver assistance of vehicles

[1306] The device provides a function to automatically control vehicle operation based on the generated parking trajectory. It also supports the driver by providing parking guidelines and voice guidance when driving independently. This function displays parking guidelines on a smartphone or in-car display to help the driver park according to them.

[1307] 6. Means for running artificial intelligence algorithms

[1308] The server uses AI technology to analyze images of the surrounding parking lot and identify the location of available spaces and obstacles. This AI algorithm uses a deep learning model to suggest the optimal parking strategy based on the results of the image analysis.

[1309] 7. A method for generating parking guidelines and displaying them on the screen

[1310] The device generates guidelines for the driver to follow based on the recognized parking information, displays them on a smartphone or in-car display, and uses voice synthesis to provide driving instructions at the appropriate time.

[1311] Specific examples

[1312] For example, a user arrives at a small parking lot in central Tokyo and launches a parking support app on their smartphone. The app scans the entire parking lot using a camera and calculates the optimal parking route using an AI algorithm. The user parks their car according to the guidelines displayed on the smartphone screen and uses voice guidance to perform the appropriate operations, completing the parking process with peace of mind. An example of a prompt that can be applied at this time is, "Please recognize the parking lot layout and identify the location of available spaces and obstacles."

[1313] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1314] Step 1:

[1315] Input: Real-time image data from a parking lot camera.

[1316] Processing: The server uses the parking lot cameras to capture images of the surrounding area, including footage of the parking space and any obstacles in the area.

[1317] Output: Captured parking lot image data.

[1318] Step 2:

[1319] Input: Parking rules, available space information, and image data.

[1320] Processing: The server collects information about parking rules and available spaces from a database and compares it with the captured image data, for example, identifying no-parking areas and the location of obstacles.

[1321] Output: Parking lot layout information as a result of image processing (available parking areas, empty spaces, obstacle locations).

[1322] Step 3:

[1323] Input: Vehicle size and specifications (length, width, height, axle distance).

[1324] Processing: The user uses a device (smartphone) to input vehicle information, which is then sent to the server.

[1325] Output: Transmitted vehicle information data.

[1326] Step 4:

[1327] Input: Parking lot layout information, vehicle information.

[1328] Processing: The server uses AI algorithms to generate the optimal parking trajectory based on the input vehicle information and parking lot layout information, for example, calculating the parking space entry angle and backing timing.

[1329] Output: Generated parking trajectory data.

[1330] Step 5:

[1331] Input: Parking trajectory data.

[1332] Processing: Based on the generated optimal parking trajectory, the device generates parking guidelines to assist the driver and displays them on a smartphone or in-car display. It also uses the voice guidance function to provide driving instructions at the appropriate time.

[1333] Output: Parking guideline display data and voice guidance data provided to the user.

[1334] Step 6:

[1335] Input: User action to prepare for delivery.

[1336] Processing: When the user begins preparing to leave, the server again captures images of the surroundings and uses AI algorithms to generate an optimal exit trajectory, taking into account any new obstacles or changing parking lot conditions.

[1337] Output: Generated outbound trajectory data.

[1338] Step 7:

[1339] Input: Outbound trajectory data.

[1340] Processing: Based on the generated departure trajectory, the terminal generates departure guidelines to assist the driver and displays them on a smartphone or in-car display. It also uses a voice guidance function to provide departure instructions at the appropriate time.

[1341] Output: Display data and voice guidance data for delivery guidelines provided to the user.

[1342] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1343] The present invention is a system that assists drivers in parking in tight spaces safely and efficiently, and further combines it with an emotion engine that recognizes the user's emotions. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automatic driving or driver assistance, as well as analyzing the user's emotions.

[1344] Explanation of program processing

[1345] 1. Acquiring surrounding images and understanding parking lot rules

[1346] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[1347] 2. Enter your car information

[1348] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[1349] 3. Parking Trajectory Generation

[1350] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[1351] 4. Implementation or assistance of automated driving

[1352] In Autonomous mode:

[1353] The device will then perform autonomous driving based on the generated optimal trajectory. In this mode, the device will automatically control steering, acceleration, and deceleration to complete the parking operation.

[1354] In driver assistance mode:

[1355] The device displays the generated optimal trajectory on an in-car monitor and assists the user in driving by following the displayed guidelines. The device also prompts the user to take action at the appropriate time using audio and visual alerts.

[1356] 5. User Emotion Recognition

[1357] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling stressed or anxious, that emotion is conveyed to the system.

[1358] 6. Emotion-based system adjustment

[1359] The server adjusts the system's behavior based on the recognized user's emotions.

[1360] If the user is feeling stressed, the device will play relaxing music and gently guide them through operations.

[1361] If the user feels anxious or nervous, the device will preferentially select the driving assistance mode and provide detailed guidance on the parking trajectory.

[1362] 7. New trajectory generation when leaving the warehouse

[1363] After parking, when the user starts preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. Based on the input information, it generates the optimal trajectory for leaving the parking lot.

[1364] 8. Conducting or assisting with deliveries

[1365] In Autonomous mode:

[1366] The terminal automatically drives the vehicle according to the generated departure trajectory and causes the vehicle to leave the parking lot.

[1367] In driver assistance mode:

[1368] The terminal displays the exit trajectory on the in-car monitor and guides the user through the exit process by following the displayed guidelines, using audio and visual alerts to encourage appropriate operation.

[1369] Specific examples

[1370] Specific example of parking

[1371] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly. The device also recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[1372] Specific example of shipping

[1373] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area and generates a new optimal trajectory for leaving. The device then either leaves the vehicle in autonomous driving mode or displays the trajectory in driver assistance mode and helps the user leave the vehicle accordingly. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[1374] The processing flow will be explained below.

[1375] Step 1:

[1376] The server acquires current surrounding images from a camera installed at the entrance of the parking lot, which are used to recognize the layout of parking spaces and the location of obstacles.

[1377] Step 2:

[1378] The server collects information on parking rules and available spaces from the database, which allows users to know the exact location of available spaces and important points to note when parking.

[1379] Step 3:

[1380] The terminal allows the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.) This information is stored in the terminal, eliminating the need to input it again.

[1381] Step 4:

[1382] The device sends the entered vehicle information to the server, which then receives all the data necessary for analysis.

[1383] Step 5:

[1384] The server analyzes the captured images of the surroundings and vehicle information, and uses AI algorithms to calculate factors such as approach angle, steering, and timing of backing up to generate the optimal parking trajectory.

[1385] Step 6:

[1386] The server then transmits the generated optimal parking trajectory data to the terminal, allowing the terminal to learn the specific parking method.

[1387] Step 7:

[1388] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, it uses facial expression analysis and voice analysis technology to detect whether the user is feeling stressed or anxious.

[1389] Step 8:

[1390] When the emotion engine recognizes emotions such as stress or anxiety, the device automatically adjusts system behavior, including playing relaxing music and providing more detailed operating instructions.

[1391] Step 9:

[1392] When parking in autonomous driving mode based on the received parking trajectory, the device automatically performs steering, acceleration, and deceleration, thereby completing parking safely and efficiently.

[1393] Step 10:

[1394] When the device is in driver assistance mode, the generated parking trajectory is displayed on the in-car monitor. The user drives and parks according to this display. The device prompts the user to operate the vehicle at the appropriate time using audio and visual alerts.

[1395] Step 11:

[1396] After parking is complete, the terminal notifies the user, allowing the user to confirm that parking was completed successfully.

[1397] Step 12:

[1398] To start preparation for delivery, the user issues an instruction to the system, which makes the server ready to acquire new information.

[1399] Step 13:

[1400] The server then acquires images of the surrounding area again to obtain the latest information, thereby ensuring safety when the vehicle leaves the warehouse.

[1401] Step 14:

[1402] The server generates an optimal trajectory for the departure, calculating a new departure route and taking into account the location of obstacles and the movement of other vehicles.

[1403] Step 15:

[1404] The server then transmits the generated retrieval trajectory data to the terminal, which then retrieves the vehicle based on this information.

[1405] Step 16:

[1406] The terminal either drives automatically along the departure trajectory or displays the information on a monitor to assist the user in leaving the parking lot. In the autonomous driving mode, the terminal performs all operations automatically. In the driver assistance mode, the user operates the device themselves.

[1407] Step 17:

[1408] After the vehicle has left the parking lot, the terminal notifies the user, allowing the user to confirm that the vehicle has safely left the parking lot.

[1409] Example 2

[1410] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1411] Parking safely and efficiently in tight parking spaces is a challenging task for many drivers. The stress and anxiety drivers feel during parking also impact safety. However, current automated driving and driver assistance systems lack the ability to recognize user emotions and adapt to the user's psychological state during the parking process. Therefore, there is a need for a parking assistance system that can adjust its behavior based on the user's emotions.

[1412] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1413] In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting the size and specifications of the vehicle, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for recognizing the user's emotions, and means for adjusting the system operation based on the recognized user's emotions, thereby making it possible to provide optimal parking support while taking the user's emotions into consideration.

[1414] "Means for acquiring images of the surrounding area" refers to technology that collects real-time images of the surrounding area using cameras and sensors installed at the entrance to the parking lot.

[1415] "Means for obtaining parking lot rules" refers to technology for obtaining traffic rules and available space information within a parking lot from a database.

[1416] "Means for inputting vehicle dimensions and specifications" refers to technology that provides an interface for users to input the dimensions and specifications (e.g., length, width, height, axle distance, etc.) of their vehicle.

[1417] "Means for generating optimal parking trajectories" refers to technology that uses AI algorithms to calculate entry angles, steering operation, timing of backing up, etc. based on acquired surrounding images and vehicle information to generate optimal parking trajectories.

[1418] "Means for autonomously driving a vehicle or assisting the driver" refers to technology that automatically controls the vehicle's steering, acceleration, and deceleration based on the generated parking trajectory, or technology that displays the trajectory on an in-vehicle monitor and assists the driver with audio and visual alerts.

[1419] "Means for recognizing user emotions" refers to an emotion engine that analyzes emotions such as stress and anxiety from the user's facial expressions and voice.

[1420] "Means for adjusting system behavior" refers to functions that change the behavior of the system based on the recognized user's emotions, such as playing relaxing music or providing gentler operating guidance.

[1421] This invention combines a system that assists drivers in parking in tight spaces safely and efficiently with an emotion engine that recognizes the user's emotions. The system includes multiple means for acquiring surrounding images, understanding parking rules, inputting vehicle information, generating optimal parking trajectories, and performing automatic driving or driver assistance, as well as analyzing the user's emotions.

[1422] The detailed system configuration uses the following hardware and software:

[1423] Acquiring surrounding images and understanding parking rules

[1424] The server obtains current surrounding images from a camera installed at the entrance to the parking lot. This high-resolution camera can clearly recognize obstacles and available spaces within the parking lot. Image analysis is performed in real time using image recognition software such as OpenCV. At the same time, the server collects information on parking lot rules and available spaces from a database. This database records traffic rules within the parking lot and space usage status.

[1425] Enter your car information

[1426] The device provides an interface for the user to input vehicle dimensions and specifications (e.g., length, width, height, and axle distance). This interface may take the form of a smartphone app or an in-vehicle display. The input information is stored on the device and sent to a server to avoid having to enter it again.

[1427] Parking trajectory generation

[1428] The server performs data analysis based on the acquired surrounding images and vehicle information. AI algorithms (e.g., TensorFlow and PyTorch) are used for the analysis, calculating factors such as approach angle, steering operation, and timing of backing up, and generating the optimal parking trajectory. The generated trajectory information is sent to the device.

[1429] Implementation or assistance of automated driving

[1430] In autonomous driving mode, the device automatically drives the vehicle based on the generated optimal trajectory. In this mode, the device automatically controls steering, acceleration, and deceleration, and completes parking using dedicated autonomous driving software (e.g., Autoware).

[1431] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. The device uses audio and visual alerts to prompt the user to take action at the appropriate time.

[1432] User Emotion Recognition

[1433] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. Emotion analysis utilizes facial and voice analysis software such as Microsoft Azure Face API and Google Speech-to-Text. For example, if the user is feeling stressed or anxious, that emotion is conveyed to the system.

[1434] Emotion-based system adjustment

[1435] The server adjusts the system's behavior based on the user's recognized emotions. Specifically, if the user feels stressed, the device will play relaxing music (e.g., using Spotify or Apple Music APIs) and provide gentle guidance on operations. If the user feels anxious or tense, the device will prioritize driving assistance mode and provide detailed guidelines.

[1436] New trajectory generation when leaving the warehouse

[1437] After parking, when the user begins preparing to leave, the server again acquires images of the surrounding area to ascertain the latest situation. Based on the input information, an AI algorithm is used to generate the optimal trajectory for leaving the parking lot and send it to the terminal.

[1438] Conducting or assisting with shipping

[1439] In autonomous driving mode, the device automatically drives the vehicle according to the generated exit trajectory and leaves the parking lot. In driver assistance mode, the device displays the exit trajectory on the in-car monitor and helps the user leave the parking lot by following the displayed guidelines. Audio and visual alerts are also used during this process.

[1440] Specific examples

[1441] As a concrete example of parking, when a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device that receives this information either parks in autonomous driving mode or displays the trajectory on a monitor in driver assistance mode, helping the user park accordingly. At the same time, the device recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[1442] Example prompts to input to the generative AI model:

[1443] "A user arrives at a parking lot, and the system automatically finds a parking space and assists the user. If the user is emotionally stressed, how will the system respond?"

[1444] For example, when the user prepares to leave the parking lot after parking, the server acquires images of the surrounding area again and generates a new optimal trajectory for leaving. The device then either leaves the parking lot in autonomous driving mode or displays the trajectory in driver assistance mode and assists the user in leaving the parking lot. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[1445] Example prompts to input to the generative AI model:

[1446] "When the user attempts to leave the parking lot after parking, please explain how the system will assist the user in leaving the parking lot and how it will respond depending on the user's emotions."

[1447] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1448] Step 1: Acquire surrounding images and understand parking rules

[1449] The server obtains real-time images of the surrounding area from a camera installed at the entrance to the parking lot. The camera image is used as input, and the obtained image data is obtained as output. This image data is analyzed to recognize the layout of parking spaces and the location of obstacles. Image recognition software such as OpenCV is used for image analysis. At the same time, the server obtains information on parking lot rules and available spaces from a database. The information registered by the user at the parking lot is used as input, and the obtained parking lot rule data is obtained as output.

[1450] Specific behavior:

[1451] The server receives the video stream from the camera.

[1452] The server performs image analysis and recognizes parking spaces and obstacles.

[1453] The server retrieves the parking rules from the database.

[1454] Step 2: Enter your vehicle information

[1455] The terminal allows the user to input the vehicle size and specifications (length, width, height, axle distance, etc.). The user's input data is used as input, and stored vehicle information is obtained as output. This information is saved on the terminal and later sent to the server. The stored data is reused to avoid having to enter the information again.

[1456] Specific behavior:

[1457] The device displays the vehicle information input screen.

[1458] The user inputs the vehicle size and specifications.

[1459] The device saves the input data and sends it to the server.

[1460] Step 3: Generate parking trajectories

[1461] The server performs data analysis based on the acquired surrounding images and the user's vehicle information. Surrounding image data and vehicle information data are used as input, and parking trajectory data is generated as output. Here, AI algorithms (e.g., TensorFlow and PyTorch) are used to calculate factors such as approach angle, steering operation, and timing of backing up, and generate the optimal parking trajectory.

[1462] Specific behavior:

[1463] The server acquires surrounding images and vehicle information.

[1464] The server analyzes using an AI algorithm.

[1465] The server generates the optimal parking trajectory and sends it to the device.

[1466] Step 4: Implementing or Assisting Autonomous Driving

[1467] In autonomous driving mode, the device performs autonomous driving based on the generated optimal trajectory. Parking trajectory data is used as input, and the parking completion status is obtained as output. Dedicated autonomous driving software (e.g., Autoware) is used.

[1468] Specific behavior:

[1469] The terminal receives the optimal orbit data.

[1470] The device begins autonomous driving using Autoware.

[1471] The device controls the steering wheel, accelerator, and brakes.

[1472] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user to drive according to the guideline. The input is the parking trajectory data, and the output is driving assistance for the user. The device uses audio and visual alerts.

[1473] Specific behavior:

[1474] The terminal displays the optimal orbit data on the monitor.

[1475] The user drives according to the guidelines.

[1476] The device assists the user with voice alerts.

[1477] Step 5: Recognizing User Emotions

[1478] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. The user's facial expression and voice data are used as input, and the recognized emotion data is obtained as output. Emotion analysis uses Microsoft Azure Face API and Google Speech-to-Text, among others.

[1479] Specific behavior:

[1480] The device uses a camera and microphone to collect the user's facial expressions and voice.

[1481] The device analyzes using an emotion engine.

[1482] The device sends the analysis results to the server.

[1483] Step 6: Adjusting the system based on emotions

[1484] The server adjusts the system's behavior based on the recognized user emotion. Emotional data is used as input and the adjusted system behavior is obtained as output. If the user is feeling stressed, the device will play relaxing music (using Spotify or Apple Music APIs) or provide gentle guidance.

[1485] Specific behavior:

[1486] The server processes the emotion data.

[1487] The device will select and play the appropriate music.

[1488] The device provides a friendly audio guide.

[1489] Step 7: New trajectory generation for exit

[1490] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. The surrounding image data is used as input, and the optimal trajectory data for leaving is obtained as output. The optimal trajectory for leaving is generated using an AI algorithm and sent to the terminal.

[1491] Specific behavior:

[1492] The user begins preparation for delivery.

[1493] The server acquires surrounding images from the camera.

[1494] The server generates the optimal trajectory for delivery and sends it to the terminal.

[1495] Step 8: Perform or assist with shipping

[1496] In the automatic driving mode, the terminal automatically drives according to the generated retrieval trajectory. The retrieval trajectory data is used as input, and the retrieval completion status is obtained as output.

[1497] Specific behavior:

[1498] The terminal receives the delivery trajectory data.

[1499] The terminal will execute the retrieval in automatic operation mode.

[1500] In driver assistance mode, the device displays the exit trajectory on the in-vehicle monitor and assists the user in following the guidelines when leaving the vehicle. The exit trajectory data is used as input, and driving assistance is provided to the user as output. Audio and visual alerts encourage appropriate operation.

[1501] Specific behavior:

[1502] The terminal displays the delivery trajectory data on the monitor.

[1503] The user issues goods according to the guidelines.

[1504] The device will prompt you with an audio alert.

[1505] (Application example 2)

[1506] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1507] Modern parking lots often have narrow spaces and complex layouts, making parking difficult for drivers. The stress and anxiety drivers feel when parking can negatively impact driving safety and efficiency. Even in these situations, there is a demand for systems that support safe and efficient parking. Furthermore, systems must provide support not only when parking but also when exiting the parking lot safely and efficiently.

[1508] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring surrounding images, means for acquiring parking lot rules, means for inputting vehicle size and specifications, means for generating an optimal parking trajectory based on the acquired images and vehicle information, means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory, means for analyzing user emotions, and means for adjusting system operation based on the user emotions. This enables the driver to park safely and efficiently, and by providing support according to the driver's emotions, it becomes possible to reduce stress and anxiety.

[1509] The "surrounding image" is an image acquired using a device such as a camera to recognize the layout of the parking lot and the location of obstacles.

[1510] "Parking rules" are rules and instructions regarding parking spaces, prohibited areas, and how to pass through a parking lot.

[1511] "Vehicle size and specifications" refers to information that refers to the physical dimensions and characteristics of a vehicle, such as length, width, height, and axle distance.

[1512] A "parking trajectory" is a planned path for a vehicle to park in a parking space, including the optimal approach angle, steering operation, and timing of backing up.

[1513] "Autonomous driving" is the ability of a vehicle to perform driving operations autonomously without human intervention.

[1514] "Supporting the driver" means providing assistance by providing guidelines, alerts, etc. when the driver parks or leaves the vehicle by themselves.

[1515] "User's emotions" refer to psychological states such as stress, anxiety, joy, etc. that are recognized from the user's facial expressions, voice, etc.

[1516] "Analyzing emotions" refers to the process of analyzing the user's facial expressions and voice to identify their emotions.

[1517] "Adjusting the system's behavior" means changing the system's functions and behavior in response to the user's emotions, for example, by playing relaxing music or providing gentle guidance.

[1518] This invention relates to a system that assists drivers in parking in narrow parking spaces safely and efficiently. The system includes multiple means for capturing images of the surrounding area, understanding parking rules, inputting vehicle information, generating an optimal parking trajectory, and performing automated driving or driver assistance and user emotion analysis.

[1519] The server acquires current surrounding images from a camera installed at the entrance of the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. At the same time, the server collects information on parking lot rules and available spaces from a database.

[1520] The terminal prompts the user to input the vehicle's size and specifications (length, width, height, axle distance, etc.). This information is stored in the terminal, eliminating the need to input it again. The server generates the optimal parking trajectory based on the captured surrounding images and vehicle information. The server uses an AI algorithm (e.g., TensorFlow) to calculate factors such as approach angle, steering operation, and timing for backing up, and generates the optimal parking trajectory.

[1521] In autonomous driving mode, the device drives autonomously based on the generated optimal trajectory. In this mode, the device automatically controls steering, acceleration, and deceleration to complete parking. In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. The device uses audio and visual alerts to prompt the user to operate at the appropriate time.

[1522] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling stressed or anxious, that emotion is communicated to the system. The server adjusts the system's operation based on the recognized user emotion. If the user is feeling stressed, the device may play relaxing music or gently guide the user through the operation. If the user is feeling anxious or tense, the device may prioritize driving assistance mode and provide detailed guidance on the parking trajectory.

[1523] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area and ascertains the latest situation. Based on the input information, it generates an optimal trajectory for leaving. In autonomous driving mode, the device automatically drives the vehicle according to the generated trajectory and leaves the parking lot. In driver assistance mode, the device displays the trajectory on the in-car monitor and assists the user in leaving the parking lot by following the displayed guidelines. During this process, the device also recognizes the user's emotions and provides appropriate support based on those emotions.

[1524] Specific examples

[1525] When a user parks their car at the entrance to a parking lot, the server captures images of the surrounding area via a camera. The server calculates the optimal parking trajectory based on information about available parking spaces and the user's vehicle information. The device receives this information and parks in autonomous driving mode, or displays the trajectory on a monitor in driver assistance mode to help the user park accordingly. The device also recognizes the user's emotions and supports them by providing appropriate music or guidance if they are feeling stressed or anxious.

[1526] Prompt Sentence Examples

[1527] "A user is trying to park a car in a tight parking space. Implement a system that recognizes whether the user is stressed or not from their facial expression, and generates an optimal parking trajectory to assist the user in parking in real time."

[1528] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1529] Step 1:

[1530] The server obtains current surrounding images from a camera installed at the entrance to the parking lot. This image is used to recognize the layout of parking spaces and the location of obstacles. The video data obtained from the camera (input) is used to identify the location of vacant spaces in the parking lot and obstacles (example of data processing: object detection using image processing algorithms). The output becomes map information of the parking lot.

[1531] Step 2:

[1532] The server collects information about parking rules and available spaces from a database. This information (input) includes rules and instructions about permitted parking spaces, prohibited areas, and how to navigate. The output is the optimal parking location that complies with the permitted parking areas and rules. This is achieved by executing a database query to obtain the required information.

[1533] Step 3:

[1534] The terminal allows the user to input the vehicle size and specifications (e.g. length, width, height, axle distance, etc.). This information (input) is stored in a vehicle database. The (output) is the complete vehicle dimensions, held in a reusable format. Input is received through a user interface.

[1535] Step 4:

[1536] The server generates an optimal parking trajectory based on the acquired surrounding images and vehicle information. It uses an AI algorithm (e.g., TensorFlow) to calculate the approach angle, steering operation, timing of backing up, etc. (example of data calculation: application of a trajectory prediction model using a neural network) and generates the generated parking trajectory (output).

[1537] Step 5:

[1538] In autonomous driving mode, the device performs autonomous driving based on the generated optimal trajectory. It automatically controls steering, acceleration, and deceleration (vehicle operation using a control algorithm) and completes parking (output). The input is optimal trajectory information. Sensor information is fed back in real time to control the vehicle's movement.

[1539] Step 6:

[1540] In driver assistance mode, the device displays the generated optimal trajectory on the in-car monitor and assists the user in driving by following the displayed guidelines. It uses audio and visual alerts to prompt the user to operate the vehicle at the appropriate time (input is the generated parking trajectory information, output is guideline display and alerts) to complete parking. It generates audio and visual guides and displays them on the user interface.

[1541] Step 7:

[1542] The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. It analyzes the user's facial images and voice data (input) (example of data processing: facial expression and voice analysis using machine learning models) and identifies the user's state of stress or anxiety (output). It then identifies the user's emotional state based on the results of the data analysis.

[1543] Step 8:

[1544] The server adjusts the system's behavior based on the recognized user emotion. If the user feels stressed, the device plays relaxing music (input: emotion recognition result, output: music playback) and gently guides the user through the operation. If the user feels anxious or tense, the device prioritizes driving assistance mode and provides more detailed guidance (output), adjusts system settings, and provides appropriate feedback.

[1545] Step 9:

[1546] After parking is complete, when the user begins preparing to leave, the server again acquires images of the surrounding area to grasp the latest situation. The acquired image data (input) is used to re-recognize the surrounding environment (example of data processing: re-application of object detection algorithm) and generate the optimal trajectory for leaving (output). The optimal exit route is calculated through data analysis.

[1547] Step 10:

[1548] In autonomous driving mode, the terminal automatically drives the vehicle according to the generated departure trajectory and leaves the parking lot. It performs autonomous driving control based on the departure trajectory information (input) and optimizes the vehicle's movement (output). It executes departure operations including real-time feedback control.

[1549] Step 11:

[1550] In driver assistance mode, the device displays the exit trajectory on the in-car monitor and assists the user in following the displayed guidelines to complete the exit. Audio and visual alerts are used to prompt the user to take the appropriate action and complete the exit (input is the exit trajectory information, output is the guideline display and alerts). The device makes full use of the user interface to provide appropriate guidance and feedback.

[1551] Example prompt: "A user is trying to park a car in a narrow parking space. Implement a system that recognizes whether the user is stressed from their facial expression and generates an optimal parking trajectory to assist the user in parking in real time."

[1552] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1553] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1554] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1555] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1556] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1557] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1558] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1559] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1560] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1561] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1562] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1563] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1564] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1565] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1566] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1567] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1568] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1569] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1570] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1571] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1572] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1573] The following is further disclosed regarding the above embodiment.

[1574] (Claim 1)

[1575] means for acquiring an image of the surroundings;

[1576] A means of obtaining parking rules;

[1577] a means of inputting vehicle size and specifications;

[1578] A means for generating an optimal parking trajectory based on the acquired image and vehicle information;

[1579] A means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory;

[1580] A system including:

[1581] (Claim 2)

[1582] 2. The system of claim 1, further comprising means for re-acquiring an image of the surroundings when leaving the parking lot and generating an optimal trajectory for leaving.

[1583] (Claim 3)

[1584] 2. The system according to claim 1, further comprising a means for displaying the generated parking trajectory and exit trajectory on an in-vehicle monitor to assist the driver.

[1585] "Example 1"

[1586] (Claim 1)

[1587] A means for acquiring surrounding conditions;

[1588] A means for obtaining parking lot rule information;

[1589] means for inputting vehicle details;

[1590] A means for generating an optimal parking route based on the acquired images and vehicle information;

[1591] A means for automatically operating the vehicle or assisting the driver based on the generated parking route;

[1592] A system including:

[1593] (Claim 2)

[1594] 2. The system according to claim 1, further comprising means for acquiring surrounding conditions again when leaving the parking lot and generating an optimal route for leaving.

[1595] (Claim 3)

[1596] 2. The system according to claim 1, further comprising means for displaying the generated parking route and exit route on an in-vehicle display device to assist the driver.

[1597] "Application Example 1"

[1598] (Claim 1)

[1599] means for acquiring an image of the surroundings;

[1600] A means of obtaining parking rules;

[1601] a means of inputting vehicle size and specifications;

[1602] A means for generating an optimal parking trajectory based on the acquired image and vehicle information;

[1603] A means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory;

[1604] means for executing an artificial intelligence algorithm to recognize the acquired image of the surroundings of the parking lot;

[1605] A means for generating parking guidelines based on the recognized position information of vac...

Claims

1. means for acquiring an image of the surroundings; A means of obtaining parking rules; a means of inputting vehicle size and specifications; A means for generating an optimal parking trajectory based on the acquired image and vehicle information; A means for automatically driving the vehicle or assisting the driver based on the generated parking trajectory; A system including:

2. 2. The system of claim 1, further comprising means for re-acquiring an image of the surroundings when leaving the parking lot and generating an optimal trajectory for leaving.

3. 2. The system according to claim 1, further comprising means for displaying the generated parking trajectory and exit trajectory on an in-vehicle monitor to assist the driver.

Images