Electronic device for acquiring external object state information, method, and storage medium

Abstract

The present application relates to an electronic device, a method and a storage medium for acquiring external object state information. The electronic device can include a camera, a memory and a processor. The processor can be configured to acquire an image using the camera, acquire spatial information of an external space at the time of acquiring the image using the image, acquire state information of an object included in the external space using the spatial information, execute a first language model using the state information to acquire a first text for describing a motion of the object, and execute a second language model using the first text and the image to acquire a second text for describing the image.

Description

Technical Field

[0001] This application relates to an electronic device, method, and non-transitory computer-readable storage medium for acquiring information related to the state of an external object using an artificial intelligence model. Background Technology

[0002] Electronic devices and / or services based on object recognition technology are being developed, including algorithms for reasoning about external objects captured in images. For example, object recognition technology can identify one or more external objects (e.g., pedestrians, vehicles) from an image. Information about the identified one or more external objects can be used to automate and / or replace user actions based on these objects, such as in automated driving.

[0003] The above information is provided as related art to aid in understanding this application. No claim is made or any decision is made as to whether any of the above content is applicable as prior art related to this application. Summary of the Invention

[0004] An electronic device is provided. The electronic device may include a camera. The electronic device may include a memory. The electronic device may include a processor. The processor may be configured to: acquire an image using the camera. The processor may be configured to: acquire spatial information about an external space at the time the image is acquired using the image. The processor may be configured to: acquire state information about a target object contained in the external space using the spatial information. The processor may be configured to: execute a first language model using the state information to acquire first text describing the motion of the target object. The processor may be configured to: execute a second language model using the first text and the image to acquire second text describing the image.

[0005] A method is provided. The method can be performed by an electronic device including a camera. The method may include: acquiring an image using the camera; acquiring spatial information about an external space at the time the image was acquired using the image; acquiring state information about a target object contained in the external space using the spatial information; performing a first language model using the state information to obtain first text describing the motion of the target object; and performing a second language model using the first text and the image to obtain second text describing the image.

[0006] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium can store one or more programs. When executed by a processor of an electronic device including a camera, the one or more programs enable the electronic device to: acquire images using the camera; acquire spatial information about an external space at the time the images are acquired using the images; acquire state information about a target object contained in the external space using the spatial information; execute a first language model using the state information to acquire first text describing the motion of the target object; and execute a second language model using the first text and the images to acquire second text describing the images. Attached Figure Description

[0007] Figure 1 A language model for obtaining information about the state of a target object, based on comparative examples, is shown.

[0008] Figure 2 This is a schematic view of an exemplary electronic device.

[0009] Figure 3 An object located outside the vehicle is shown.

[0010] Figure 4 This is a flowchart illustrating the operation of an electronic device for acquiring information related to the state of a target object.

[0011] Figure 5 It is a diagram of a system for describing electronic devices that acquire information related to the state of a target object.

[0012] Figure 6An exemplary operation of an electronic device for obtaining data for vehicle control from the status information of a target object is shown.

[0013] Figure 7 An exemplary operation of an electronic device is shown, which retrieves data from the status information of a target object to inform the user of anomalies.

[0014] Figure 8 This is a flowchart of the operation of an electronic device that uses a language model to obtain text related to the state of a target object.

[0015] Figure 9 It is a graph used to describe the data exchange between language models.

[0016] Figure 10 A block diagram example of a vehicle autonomous driving system according to one embodiment is shown.

[0017] Figure 11 and Figure 12 This is a block diagram example illustrating an autonomous driving mobile body according to one embodiment.

[0018] Figure 13 This illustrates examples of gateways associated with user devices according to various embodiments.

[0019] Figure 14 This is a schematic diagram illustrating the operation of an electronic device for training a neural network based on a training dataset, according to one embodiment.

[0020] Figure 15 This is a block diagram of an electronic device according to one embodiment. Detailed Implementation

[0021] The specific structural or functional descriptions of embodiments of the invention disclosed in this specification are merely illustrative of embodiments of the invention, which may be implemented in various forms and are not limited to those described in this specification.

[0022] Since the embodiments of the present invention can be modified and take many forms, embodiments will be shown in the accompanying drawings and described in detail in this specification. However, this is not intended to limit the embodiments of the present invention to a particular disclosed form, but should be understood to include modifications, equivalents, or substitutions that fall within the spirit and technical scope of the present invention.

[0023] While terms such as "first" or "second" may be used to describe various constituent elements, the constituent elements should not be limited by these terms. These terms are used only to distinguish one constituent element from another; for example, without departing from the scope of the invention, a first constituent element may be named a second constituent element, and similarly, a second constituent element may be named a first constituent element.

[0024] When it is said that one constituent element is "connected" or "accessed" to another constituent element, it should be understood that it can be directly connected to or accessed to that other constituent element, but there may be other constituent elements in between. In contrast, when it is said that one constituent element is "directly connected" or "directly accessed" to another constituent element, it should be understood that there are no other constituent elements in between. Expressions describing the relationship between constituent elements, such as "between...", "immediately between...", or "directly adjacent to...", should also be interpreted in the same way.

[0025] The terminology used in this specification is for illustrative purposes only and is not intended to limit the invention. Unless the context clearly requires otherwise, singular expressions include plural expressions. In this specification, terms such as "comprising" or "having" are intended to indicate the presence of the listed features, numbers, steps, actions, constituent elements, components, or combinations thereof, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, actions, constituent elements, components, or combinations thereof.

[0026] Unless otherwise defined, all terms used herein, including technical and scientific terms, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant technical context and shall not be interpreted as having an idealized or overly formal meaning unless expressly defined herein.

[0027] The embodiments will now be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals given in the figures may denote the same configurations, and repeated descriptions may be omitted.

[0028] Figure 1 A language model for obtaining information about the state of a target object, based on comparative examples, is shown.

[0029] Reference Figure 1 Electronic devices (e.g.) Figure 2The electronic device 200 may include a system 100 for providing information about an external object to a user. System 100 may include a language model 120. The electronic device may acquire output data 130 from an image 110 via the driving system 100. The image 110 may be viewed through a camera of the electronic device (e.g., a camera). Figure 2 (Acquired by camera 230).

[0030] Electronic device 200 can perform operations on language model 120 based on image 110 to generate output data 130. Language model 120 may include multiple models. For example, language model 120 may include a first model 121 and a second model 122. For example, the first model 121 may be referred to as a pre-trained model. For example, the second model 122 may be referred to as a model that has been additionally trained on parameters used to predict the motion of object 111 corresponding to an external object in image 110. The additionally trained parameters of the second model 122 may be applied to the first model 121.

[0031] The output data 130 can take many forms. For example, the output data 130 may include images and / or text used by the system 100 to inform the user of the predicted motion of the object 111. For example, the output data 130 may be displayed on the screen of an electronic device, or may be transmitted through the speaker of the electronic device (e.g., Figure 2 The speaker output is 240.

[0032] System 100 can predict the motion of external objects using a language model 120 that can be further trained. However, since the language model 120 needs to be trained independently for each external object, the cost of building system 100 to predict the motion of said external objects may increase. Therefore, a new system for predicting (or recognizing) the motion of external objects may be needed.

[0033] Figure 2 This is a schematic view of an exemplary electronic device.

[0034] The electronic device 200 (or external electronic device) of the various embodiments disclosed herein can be of various forms. The electronic device 200 may include, for example, a computer device, a portable multimedia device, a camera (e.g., a dashcam), a wearable device, a server, or a home appliance. The electronic device 200 (or external electronic device) of the embodiments herein is not limited to the devices described above.

[0035] Reference Figure 2 This may include items contained in the vehicle (e.g., Figure 3The electronic device 200 is located in the vehicle 310. For example, the electronic device 200 may be installed in the vehicle 310. For example, the electronic device 200 may correspond to a device attached to the vehicle 310 (e.g., a dashcam), or may be included in said device. However, this application is not limited thereto. For example, the electronic device 200 may be built-in into the vehicle 310. For example, the electronic device 200 may correspond to an electronic control unit (ECU) within the vehicle 310, or may be included in the ECU. The ECU may be referred to as an electronic control module (ECM). Figure 1 The electronic device 200 may include Figure 12 At least a portion of the control device 1200, or a portion thereof, that can correspond to Figure 12 At least a portion of the control device 1200.

[0036] The electronic device 200 may include a processor 210, a memory 220, a camera 230, and at least one sensor 250.

[0037] Processor 210 may include hardware components capable of processing data using instructions stored in memory 220. The hardware components for processing data may include a central processing unit (CPU) (e.g., including processing circuitry). The hardware components for processing data may include a graphics processing unit (GPU) (e.g., including processing circuitry). The hardware components for processing data may include a display processing unit (DPU) (e.g., including processing circuitry). The hardware components for processing data may include a neural processing unit (NPU) (e.g., including processing circuitry). Processor 210 may include one or more cores. For example, processor 210 may have a multi-core processor architecture such as dual-core, quad-core, or hexa-core. As a non-limiting example, processor 210 may utilize an artificial intelligence model (e.g., a generative artificial intelligence (AI) model) to perform at least a portion of the operations described in this application.

[0038] The processor 210 can control the memory 220 of the electronic device 200 and / or one or more components (e.g., camera 230, speaker 240, and / or at least one sensor 250). For example, the processor 210 may include... Figure 12At least a portion of the processor 1224, or possibly corresponding to Figure 12 At least a portion of the processor 1224.

[0039] Memory 220 may include one or more storage media or storage mediums. For example, the one or more storage media may include hard disks, flash memory, permanent memory such as ROM (read-only memory), semi-permanent memory such as RAM (random access memory), any other suitable type of storage assembly, or any combination thereof. Memory 220 may include cache memory (e.g., may be included within processor 210), which is one or more different types of memory used for temporarily storing data for the function or feature of electronic device 200. The memory 220 may be fixedly embedded in the electronic device 200, or may be integrated into one or more suitable types of components (e.g., a subscriber identity module (SIM) card and / or a secure digital (SD) memory card) capable of being repeatedly inserted into and removed from the electronic device 200. For example, the memory 220 may include... Figure 12 At least a portion of the memory 1222, or which may correspond to Figure 12 At least a portion of the memory 1222.

[0040] Camera 230 may include one or more lenses and an image sensor. For example, the one or more lenses may be implemented as a lens assembly. The lens assembly may include a wide-angle lens or a telephoto lens. The one or more lenses may collect light around camera 230 (or around electronic device 200) to acquire an image.

[0041] For example, the image sensor within the camera 230 can convert light collected by one or more lenses into electrical signals to acquire an image. The image sensor may include, for example, one image sensor selected from image sensors with different properties, multiple image sensors having the same properties, or multiple image sensors with different properties. These image sensors with different properties may, for example, include RGB (red, green, blue) sensors, BW (black and white) sensors, IR (infrared) sensors, or UV (ultraviolet) sensors. The individual image sensors included in the image sensor may, for example, be implemented using CCD (charge-coupled device) sensors or CMOS (complementary metal oxide semiconductor) sensors.

[0042] Electronic device 200 may include a loudspeaker 240. Loudspeaker 240 may output an audio signal. Loudspeaker 240 may receive an electrical signal. Loudspeaker 240 may include elements for acquiring the electrical signal. Loudspeaker 240 may convert the electrical signal into a sound wave signal. Loudspeaker 240 may include elements for converting the electrical signal into a sound wave signal. Loudspeaker 240 may output an audio signal containing the converted sound wave signal. Loudspeaker 240 may include elements for outputting the audio signal. Loudspeaker 240 may include: at least one voice coil for providing vibration to a diaphragm within the loudspeaker 240; and a magnet for forming a magnetic field. When current flows through the at least one voice coil, the magnetic field formed by the voice coil may interact with the magnetic field formed by the magnet, causing the voice coil to vibrate. The diaphragm connected to the voice coil may vibrate based on the vibration of the voice coil. Loudspeaker 240 may output an audio signal based on the diaphragm. Loudspeaker 240 may be a non-essential component of electronic device 200. For example, the speaker 240 may be an optional component of the electronic device 200.

[0043] The at least one sensor 250 may include a sensor for identifying a vehicle containing electronic devices 200 (e.g., Figure 3The vehicle (310) may have one or more sensors for its external environment. For example, the one or more sensors may include a LiDAR sensor for emitting laser light. The LiDAR sensor can detect reflected light from an external object after the emitted laser light is reflected. The LiDAR sensor can obtain data about the distance between the vehicle and the external object and / or about the shape of the external object by detecting the reflected light. Alternatively, the one or more sensors may include a radar sensor for emitting electromagnetic waves (e.g., radio waves). The radar sensor can detect reflected waves from an external object after the emitted electromagnetic waves are reflected. The radar sensor can obtain data about the distance between the vehicle and the external object and / or about the relative speed of the vehicle relative to the external object by detecting the reflected waves.

[0044] The at least one sensor 250 may include a microphone 251 for acquiring sound information from outside the electronic device 200. For example, the microphone 251 may convert sound waves generated by an external sound source into electrical signals. For example, the microphone 251 may be a sensor for detecting (or recognizing) voice input from a user toward an external object recognized by the electronic device 200. The microphone 251 may be a non-essential component of the electronic device 200. For example, the microphone 251 may be an optional component of the electronic device 200. For example, the at least one sensor 250 may include... Figure 11 At least a portion of sensor 1210, or that may correspond to Figure 11 At least a portion of the sensor 1210.

[0045] Figure 3 An object located outside the vehicle is shown.

[0046] Reference Figure 3 Environment 301 may include vehicle 310 and external objects. For example, the external objects may include a first external object 350, a second external object 360, and a third external object 370.

[0047] Vehicle 310 may include a mobile body configured to move on the ground. For example, vehicle 310 may include an autonomous mobile body that operates without a driver, but rather according to control signals received from electronic device 200. For example, vehicle 310 may include a vehicle that travels on a road (e.g., a passenger car and / or a bus). However, it is not limited to this. For example, vehicle 310 may include special-purpose vehicles (e.g., golf carts, freight vehicles, and / or low-speed mobile vehicles). For example, vehicle 310 may include agricultural machinery (e.g., tractors and / or combine harvesters). For example, vehicle 310 may include construction machinery. For example, Figure 3 The vehicle 310 may include Figure 13At least a portion of vehicle 1300, or possibly corresponding to Figure 13 At least a portion of the 1,300 vehicles.

[0048] External objects (e.g., first external object 350, second external object 360, and third external object 370) may include various types of physical objects contained in the surrounding environment (e.g., environment 301) of vehicle 310. For example, first external object 350 and second external object 360 may represent physical objects whose direction of movement and speed change (e.g., pedestrians, bicycles, vehicles, and / or animals). For example, third external object 370 may represent stationary physical objects (e.g., signs, speed bumps, and / or obstacles). For example, external objects may be described as objects that vehicle 310 cannot physically pass through, or objects that pose a risk of vehicle body damage or rollover when passed through.

[0049] Electronic device 200 may include an artificial intelligence model for predicting the state of a target object (e.g., an external object). Electronic device 200 may utilize information acquired through the artificial intelligence model to provide various services to a user (e.g., autonomous driving, theft notification). The operation of electronic device 200 for acquiring information related to the state of the target object will refer to... Figure 4 Describe it.

[0050] Figure 4 This is a flowchart illustrating the operation of an electronic device used to acquire information related to the state of a target object.

[0051] The above operations can be performed by Figure 2 The electronic device 200 or its processor 210 is shown to perform the operations. In the following embodiments, the operations may be performed sequentially, but not necessarily in sequence. For example, the order of the operations may be changed, or at least two operations may be performed in parallel.

[0052] Reference Figure 4 In operation 410, the processor 210 may utilize a camera (e.g., Figure 2 The camera 230 acquires images. For example, the images may include at least a portion of an image based on the camera's viewpoint, or may correspond to at least a portion of the image based on the viewpoint. For example, the processor 210 may acquire images by capturing the external space of the vehicle via the camera while the vehicle containing the electronic device 200 is in motion.

[0053] In operation 420, processor 210 can acquire spatial information of the external space at the time the image is acquired (e.g., Figure 5 Spatial information 520).

[0054] Spatial information about the external space may include location information of a target object contained within the external space. For example, the location information may include information about the spatial extent occupied by the target object within the external space. As a non-limiting example, the location information may be represented as coordinate data on a three-dimensional coordinate system set with reference to the electronic device 200.

[0055] Spatial information about external space may include temporal information. For example, since processor 210 acquires data about the moment an image is captured from a camera, the spatial information may include time (or moment) information. Because the spatial information includes both location and time information, it can be understood or referred to as four-dimensional information. Since the spatial information includes temporal information, the information or data obtained using the spatial information can be processed by the system of electronic device 200 (e.g., Figure 5 The system (500) is processed in a time series manner.

[0056] Processor 210 uses images to acquire spatial information of external space, which may include operations to distinguish target objects contained within said external space. For example, processor 210 may use image segmentation techniques to identify the type of target object. For example, image segmentation techniques can be described as techniques that identify the type of target object corresponding to said object by separating the pixels of an object contained in a video (or image). For example, electronic device 200 may use software programs or AI models to perform image segmentation techniques to distinguish target objects contained within external space.

[0057] In operation 430, processor 210 can utilize spatial information to obtain state information about a target object contained in the external space (e.g., Figure 5 The target object's status information (530) is as follows. For example, the target object can be understood or referred to as a unit that distinguishes all elements (e.g., roads, sidewalks, trees) contained in the external space according to preset criteria. The target object may include a single object and / or an area of ​​the external space. For example, the single object may include a moving object (e.g., Figure 3 The first external object 350, the second external object 360) and / or stationary objects (e.g., Figure 3 The third external object 370). For example, the area may include a vehicle containing electronic devices 200 (e.g., Figure 3 Vehicles 310) can travel in areas (e.g., roads) and / or cannot travel in areas (e.g., sidewalks).

[0058] The state information may include information for predicting the motion of the target object. For example, the state information may include, but is not limited to, the target object's trajectory, its relative position to the electronic device 200, its motion, and / or its size. For example, the state information may include all information for predicting or identifying the state of the target object after the moment the image is acquired.

[0059] In operation 430, processor 210 can use spatial information to identify features of a target object. The state information acquired by processor 210 can be differentiated based on the identified features of the target object. As a non-limiting example, the features may include the shape of the target object and the speed of the target object.

[0060] The processor 210 can acquire state information about the target object based on the characteristics of the identified target object. As a non-limiting example, if the identified target object is a person, the acquired state information may include information about movement (e.g., kicking, looking at a smartphone, standing), location, posture, and movement trajectory. As a non-limiting example, if the identified target object is a vehicle, the acquired state information may include information about the vehicle's state (e.g., driving, parked, stationary), movement trajectory, and driving mode. As a non-limiting example, if the identified target object is an obstacle (e.g., a pothole, a speed bump), the acquired state information may include information about the obstacle's surface, size, and degree of danger.

[0061] In operation 440, processor 210 may use state information to execute the first language model (e.g., Figure 5 The first language model 541), to obtain the first text for describing the motion of the target object (e.g., Figure 6 The first text (650). For example, a first language model can be described as a model that predicts the state information of a target object at the next moment by tokenizing the state information of the target object. The first language model can generate first text by mapping the predicted state information to words (or language expressions).

[0062] The first language model may correspond to, or may include, a physics-informed model used to predict the state of a target object. For example, based on the state information of the target object, the first language model may process the input state information according to physical laws (e.g., laws of acceleration, laws of thermodynamics) to predict the state information (or state) of the target object at the next moment. For example, the first language model may include a simulator that models physical phenomena based on the state information of the target object. In the case of a first language model, because it uses physical laws to process the input data, it may be referred to or referenced as a Physical LLM.

[0063] In operation 450, processor 210 may use the first text and image to execute a second language model (e.g., Figure 5 The processor 210 can obtain a second language model (542) to acquire second text for describing the image. For example, the processor 210 can acquire second text through the second language model, which describes the state and / or movement of an object contained in the image and corresponding to the target object after the image acquisition time.

[0064] In operation 450, processor 210 can generate information based on the second text for providing various services to the user.

[0065] According to one embodiment, processor 210 may utilize second text to generate information for controlling a vehicle containing electronic devices 200 (e.g., Figure 3 The processor 210 can use the driving information of the vehicle 310 to control the motor and / or engine (e.g., Figure 11 The engine 1106, thereby providing users with driving-related services (e.g., autonomous driving).

[0066] According to one embodiment, processor 210 may use second text to generate guidance information for informing a user of an anomaly caused by a target object. For example, an anomaly caused by a target object may include a vehicle (e.g., caused by the target object) Figure 3 Vehicle 310) damage and vehicle theft. As a non-limiting example, in order to inform the user of the anomaly caused by the target object, the processor 210 may use the guidance information to control a speaker (e.g., [unclear]) included in the electronic device 200. Figure 2 (Speaker 240).

[0067] However, this is not the only limitation. According to one embodiment, processor 210 can generate information for providing various services to the user based on the second text and user input. For example, processor 210 can utilize the second text and user input to generate information for controlling a vehicle (e.g., Figure 3Information about the vehicle 310. The processor 210 can use this information to control the motor and / or engine (e.g., vehicle 310). Figure 11 The engine 1106), thereby providing the user with services for assisted driving (e.g., autonomous driving, automatic parking). As a non-limiting example, user input may be made by the microphone of the electronic device 200 (e.g., Figure 2 The microphone 251 was able to receive the signal.

[0068] Electronic device 200 may include an AI system for acquiring information related to the state of a target object (e.g., Figure 5 The AI ​​system will refer to System 500). Figure 5 Please provide an explanation.

[0069] Figure 5 It is a diagram of a system for describing electronic devices that acquire information related to the state of a target object.

[0070] Reference Figure 5 The electronic device 200 may include a system 500 for generating output data 550 from various inputs 510. The output data 550 may correspond to... Figure 4 At least a portion of the second text, or may include Figure 4 At least a portion of the second text.

[0071] The various inputs 510 may be referred to or associated with as multimodal inputs. The various inputs 510 may include images 511, video 512, and / or sensor data 513. For example, images 511 may include data transmitted through a camera (e.g., ...). Figure 2 Images acquired by camera 230. For example, video 512 may include a set of images captured continuously by the camera. For example, sensing data 513 may include images acquired by at least one sensor (e.g., Figure 2 Data acquired by at least one sensor 250.

[0072] Processor 210 can acquire (or generate) spatial information 520 from various inputs 510 via drive system 500. For example, processor 210 can acquire spatial information of the external space (e.g., the space outside the vehicle) corresponding to the moment when various inputs 510 are acquired via drive system 500. For example, system 500 can generate spatial information 520 by utilizing mathematical operations of a first software program, but is not limited thereto. For example, system 500 can generate spatial information 520 from various inputs 510 through operations of an AI model. For example, a description of spatial information can be found in [reference needed]. Figure 4 Description of spatial information in China.

[0073] Processor 210 can acquire state information 530 about a target object from spatial information 520 via drive system 500. For example, system 500 can generate state information by utilizing mathematical operations of a second software program, but is not limited to this. For example, system 500 can generate state information 530 about the target object from spatial information 520 through calculations of an AI model. For example, the state information 530 can be referred to as information generated by processor 210 based on spatial information 520 at the time when various inputs 510 are acquired or at the time when various inputs 510 are input (or injected) into system 500. That is, state information 530 may include time-related information or parameters. For example, a description of the state information 530 can be found in [reference needed]. Figure 4 Description of status information in the text.

[0074] System 500 may include a subsystem 540 for generating output data 550. Subsystem 540 may include a first language model 541 and a second language model 542. Since subsystem 540 includes multiple language models, it may be referred to as or referenced as an integrated model.

[0075] Electronic device 200 can acquire first data based on first language model 541 via drive system 500. For example, in system 500, state information 530 about a target object can be input (or injected) into first language model 541. System 500 can generate first data describing the state or motion of the target object from the state information 530 via first language model 541. The first data may correspond to... Figure 4 The first text. System 500 can pass the first data output from the first language model 541 to the second language model 542.

[0076] Electronic device 200 can acquire output data 550 based on second language model 542 via drive system 500. For example, in system 500, first data generated by first language model 541 can be input (or injected) into second language model 542. System 500 can generate output data 550 from the first data using second language model 542 to describe various inputs 510. The output data 550 may correspond to... Figure 4 The second text.

[0077] In system 500, there can be multiple paths for inputting (or injecting) data (or information) into the second language model. These paths may include a first path and a second path. For example, the first path may represent a path where various inputs 510 are processed sequentially through software programs and operations of the first language model 541, thereby generating first data that is input into the second language model 542. For example, the second path may represent a path where various inputs 510 are directly input into the second language model 542. However, in system 500, the second path may be a non-essential configuration. For example, the second path may be an optional configuration of system 500.

[0078] The first language model 541 included in system 500 can predict and generate the next state information (or state) of the target object based on state information about the target object. The operation of the electronic device that predicts the state of the target object through the first language model 541 will refer to... Figure 6 and Figure 7 Please provide an explanation.

[0079] Figure 6 An exemplary operation of an electronic device for obtaining data for vehicle control from the state information of a target object is shown.

[0080] Reference Figure 6 Example 611 can represent the state information S of the target object at time t. t For example, the first language model 541 can receive an example 611 containing state information of a first target 621 and a second target 622 at time t as input, and predict the state information S after n seconds from time t. t+n Example 612 can represent the state information S of the target object at time t+n. t+n Examples. Example 612 may include state information of a first target 621 and a second target 622 at time t+n. As a non-limiting example, the state information of the first target 621 may include information about the state of the first target 621 (e.g., driving, parked, stationary), its movement trajectory, and its driving mode. As a non-limiting example, the state information of the second target 622 may include information about the state of the second target 622 (e.g., driving, parked, stationary), its movement trajectory, and its driving mode.

[0081] The first language model 541 can be based on physical laws, from the state information S of the target object at time t. t Predict the state information S of the target object at time t+n t+n The first language model 541 can generate state information S to describe the predicted target object. t+nExample 612, first text 650. For example, first language model 541 can generate first text 650 by mapping the predicted state information of the first target 621 and the second target 622 contained in example 612 to words (or language expressions). First text 650 may be referred to as text used to describe the state or motion of the first target 621 and the second target 622.

[0082] Electronic device 200 can be driven by a drive system (e.g., Figure 5 The system 500 transmits first text 650 generated by the first language model 541 to the second language model 542. The first text 650 may be referred to in the system as input to the second language model 542. The electronic device 200 may perform operations on the second language model 542 based on the first text 650 to obtain second text for controlling the vehicle. For example, the electronic device 200 may use the second text to generate text for controlling the vehicle (e.g., ...). Figure 3 The vehicle 310) provides driving information to enable the vehicle to avoid the first target 621 and the second target 622.

[0083] Figure 7 An exemplary operation of an electronic device for obtaining data from the status information of a target object to inform a user of anomalies is shown.

[0084] Reference Figure 7 Example 711 can represent the state information S of the target object at time t. t For example, the first language model 541 can receive an example 711 containing the state information of a third target 721 at time t as input, and predict the state information S after n seconds from time t. t+n Example 712 can represent the state information S of the target object at time t+n. t+n Example 712 may include state information of a third target 721 at time t+n. As a non-limiting example, the state information of the third target 721 may include information about the third target 721's motion (e.g., kicking, looking at a smartphone, standing), position, posture, and movement trajectory. For example, information about the posture may be referred to as skeleton data.

[0085] The first language model 541 can be based on physical laws, from the state information S of the target object at time t. t Predict the state information S of the target object at time t+n t+n The first language model 541 can generate state information S to describe the predicted target object. t+nExample 712, first text 650. For example, first language model 541 can generate first text 650 by mapping the predicted state information of the third target 721 included in example 712 to words (or language expressions). First text 650 may be referred to as text used to describe the state or motion of said third target 721.

[0086] Electronic device 200 can be driven by a drive system (e.g., Figure 5 The system 500 transmits first text 650 generated by the first language model 541 to the second language model 542. The first text 650 may be referred to in the system as input to the second language model 542. The electronic device 200 may perform operations on the second language model 542 based on the first text 650 to obtain second text for informing the user of an anomaly (or risk) caused by a target object. For example, the electronic device 200 may use the second text to generate information for informing the user of an anomaly (or risk) caused by a third target object 721 affecting a vehicle (e.g., Figure 3 Guidance information regarding damage caused or to be caused by the vehicle (310).

[0087] In the system of electronic device 200 (e.g., Figure 5 In system 500, the first language model 541 and the second language model 542 can exchange information (or data). The operation of electronic device 200 feeding back information (or data) generated by the second language model 542 to the first language model 541 will refer to... Figure 8 and Figure 9 Please provide an explanation.

[0088] Figure 8 This is a flowchart illustrating the operation of an electronic device that uses a language model to obtain text related to the state of a target object.

[0089] Reference Figure 8 In operation 810, processor 210 may use the second text to execute the first language model (e.g., Figure 5 The first language model 541), thereby obtaining a method for reducing the first text (e.g., Figure 6 The processor 210 can calculate the similarity between the tokens (e.g., words) constituting the first text and the tokens constituting the second text using a first language model. Based on the calculated similarity, the processor 210 can identify the error (or difference) between the first text and the second text. The processor 210 can then perform operations on the first language model based on this error to obtain a third text used to guide the second language model to reduce the error between the first text and the second text.

[0090] In operation 820, processor 210 may use the third text and image to execute a second language model, thereby obtaining fourth text for describing the image. For example, the fourth text description is contained within the image (e.g., Figure 5 The accuracy of the actual action (or behavior) of the object corresponding to the target in the image 511 may be higher than the accuracy of the actual action (or behavior) of the object described in the second text. For example, the service provided to the user by the processor 210 using information generated through the fourth text (e.g., autonomous driving, theft alarm) may have improved quality compared to the service provided to the user by the processor 210 using information generated through the second text.

[0091] Operations 810 to 820 can be repeated to ensure that the text generated by the second language model accurately describes various inputs (e.g., Figure 5 Various inputs (510).

[0092] Figure 9 It is a graph used to describe the data exchange between language models.

[0093] Reference Figure 9 Subsystem 540 may include a first language model 541 and a second language model 542. Processor 210 may generate motion data 910 of the target object by executing operations on the first language model 541, based on the target object's state information (e.g., state information 530 about the target object). The motion data 910 of the target object may include... Figure 6 The first text 650. In subsystem 540, data 910 about the motion of the target object can be transferred from the first language model 541 to the second language model 542.

[0094] The processor 210 can generate parsed data 920 based on the motion data 910 of the target object by performing calculations on the second language model 542. The parsed data 920 of the motion data of the target object may include... Figure 8 The second text. In subsystem 540, data 920, which parses data about the motion of the target object, can be passed from the second language model 542 to the first language model 541.

[0095] In subsystem 540, data exchange (e.g., data 910 about the motion of the target object, data 920 parsing the data about the motion of the target object) occurring between the first language model 541 and the second language model 542 can be repeatedly performed. For example, subsystem 540 may include a feedback loop for data exchange between the first language model 541 and the second language model 542. Processor 210 can generate a description of various inputs (e.g., data 910 about the motion of the target object, data 920 about the motion of the target object) by performing operations on the second language model 542, which repeatedly exchanges data with the first language model 541 via the feedback loop. Figure 5 The text describes various inputs (510). The accuracy of the text describing the various inputs is higher than the accuracy of the text describing the various inputs generated without the feedback loop.

[0096] Figure 10 A block diagram example of a vehicle autonomous driving system according to one embodiment is shown.

[0097] according to Figure 10 The vehicle autonomous driving system 1000 may be a deep learning network including a sensor 1003, an image preprocessor 1005, a deep learning network 1007, an artificial intelligence (AI) processor 1009, a vehicle control module 1011, a network interface 1013, and a communication unit 1015. In various embodiments, the various components can be connected through different interfaces. For example, sensor data sensed and output by the sensor 1003 can be fed to the image preprocessor 1005. Sensor data processed by the image preprocessor 1005 can be fed to the deep learning network 1007 run by the AI ​​processor 1009. The output of the deep learning network 1007 run by the AI ​​processor 1009 can be fed to the vehicle control module 1011. Intermediate results of the deep learning network 1007 running on the AI ​​processor 1009 can be fed to the AI ​​processor 1009. In various embodiments, the network interface 1013 can communicate with in-vehicle electronic devices (e.g., Figure 2The system communicates with the electronic device 200 to transmit autonomous driving path information and / or autonomous driving control commands for vehicle autonomous driving to the internal module. In one embodiment, the network interface 1013 may be used to transmit sensor data acquired by sensor 1003 to an external server. In some embodiments, the autonomous driving control system 1000 may appropriately include additional or fewer components. For example, in some embodiments, the image preprocessor 1005 may be an optional component. As another example, a post-processing module (not shown) may be included in the autonomous driving control system 1000 for performing post-processing on the output of the deep learning network 1007 before providing the output to the vehicle control module 1011.

[0098] In some embodiments, sensor 1003 may include more than one sensor. In various embodiments, sensor 1003 may be mounted at different locations on the vehicle. Sensor 1003 may face more than one different direction. For example, sensor 1003 may be mounted on the front, sides, rear, and / or roof of the vehicle, facing forward-facing, rear-facing, side-facing, etc. In some embodiments, sensor 1003 may be an image sensor, such as a high dynamic range camera. In some embodiments, sensor 1003 may include non-visual sensors. In some embodiments, sensor 1003 may include radar (RADAR), lidar (LiDAR), and / or ultrasonic sensors in addition to image sensors. In some embodiments, sensor 1003 is not mounted on the vehicle having vehicle control module 1011. For example, sensor 1003 can be used as part of a deep learning system to capture sensor data and can be installed in the environment or on the road, and / or on surrounding vehicles.

[0099] In some embodiments, the image pre-processor 1005 can be used to preprocess sensor data from the sensor 1003. For example, the image pre-processor 1005 can be used to preprocess sensor data by splitting the sensor data into one or more components, and / or to post-process one or more components. In some embodiments, the image pre-processor 1005 can be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the image pre-processor 1005 can be a tone-mapper processor for processing high dynamic range data. In some embodiments, the image pre-processor 1005 can be a component of the AI ​​processor 1009.

[0100] In some embodiments, the deep learning network 1007 may be a deep learning network for implementing control commands to control an autonomous vehicle. For example, the deep learning network 1007 may be an artificial neural network, such as a convolutional neural network (CNN), trained using sensor data, and the output of the deep learning network 1007 is provided to the vehicle control module 1011.

[0101] In some embodiments, the artificial intelligence (AI) processor 1009 may be a hardware processor for running the deep learning network 1007. In some embodiments, the AI ​​processor 1009 may be a specialized AI processor for performing inference on sensor data via a convolutional neural network (CNN). In some embodiments, the AI ​​processor 1009 may be optimized for the bit depth of the sensor data. In some embodiments, the AI ​​processor 1009 may be optimized for deep learning operations, such as operations in neural networks that include convolution, inner product, vector, and / or matrix operations. In some embodiments, the AI ​​processor 1009 may be implemented using multiple graphics processing units (GPUs) capable of efficiently performing parallel processing.

[0102] In various embodiments, the AI ​​processor 1009 can be coupled to a memory storing instructions via an input / output interface. These instructions, when executed by the AI ​​processor 1009, can perform deep learning analysis on sensor data from the sensor 1003 and generate machine learning results to enable the vehicle to operate at least partially autonomously. In some embodiments, the Vehicle Control Module 1011 can process vehicle control instructions output by the AI ​​processor 1009 and translate the output of the AI ​​processor 1009 into instructions for controlling various modules of the vehicle. In some embodiments, the Vehicle Control Module 1011 can be used to control the vehicle to achieve autonomous driving. In some embodiments, the Vehicle Control Module 1011 can adjust the vehicle's steering and / or speed. For example, the Vehicle Control Module 1011 can be used to control the vehicle's movement, including operations such as deceleration, acceleration, steering, lane changing, and lane keeping. In some embodiments, the vehicle control module 1011 may generate control signals for controlling vehicle lighting, such as brake lights, turn signals, and headlights. In some embodiments, the vehicle control module 1011 may be used to control vehicle audio-related systems, such as the vehicle's sound system, vehicle's audio warnings, vehicle's microphone system, and vehicle's horn system.

[0103] In some embodiments, the vehicle control module 1011 can be used to control notification systems, including warning systems for alerting passengers and / or the driver to driving events, such as approaching a predetermined destination or a potential collision. In some embodiments, the vehicle control module 1011 can be used to adjust vehicle sensors, such as sensor 1003. For example, the vehicle control module 1011 can adjust the orientation of sensor 1003, change the output resolution and / or format type of sensor 1003, increase or decrease the capture rate, adjust the dynamic range, and adjust the focal length of the camera. Furthermore, the vehicle control module 1011 can individually or collectively enable or disable the operation of sensors.

[0104] In some embodiments, the vehicle control module 1011 can be used to modify parameters of the image preprocessor 1005, such as adjusting the frequency range of filters, adjusting edge detection parameters for feature and / or object detection, or adjusting channels and bit depth. In various embodiments, the vehicle control module 1011 can be used to control the vehicle's autonomous driving functions and / or driver assistance functions.

[0105] In some embodiments, network interface 1013 may serve as the internal interface between the modules of the autonomous driving control system 1000 and the communication unit 1015. Specifically, network interface 1013 may function as a communication interface for receiving and / or sending data including voice data. In various embodiments, network interface 1013 may connect to an external server through communication unit 1015 to enable voice call connections, receive and / or send text messages, transmit sensor data, update vehicle software to the autonomous driving system, or update the software of the vehicle's autonomous driving system.

[0106] In various embodiments, the communication unit 1015 may include multiple wireless interfaces, such as cellular or WiFi. For example, the network interface 1013 can connect to an external server via the communication unit 1015 to receive updates on operating parameters and / or instructions for the sensor 1003, image preprocessor 1005, deep learning network 1007, AI processor 1009, and vehicle control module 1011. For example, the machine learning model of the deep learning network 1007 can be updated via the communication unit 1015. In another example, the communication unit 1015 can be used to update the operating parameters (such as image processing parameters) of the image preprocessor 1005 and / or the firmware of the sensor 1003.

[0107] In other embodiments, the communication unit 1015 can be used to activate communication with emergency services and emergency contacts in the event of an accident or near-accident. For example, in a collision, the communication unit 1015 can be used to call emergency services for assistance and to notify external parties of emergency services such as collision details and vehicle location. In various embodiments, the communication unit 1015 can also be used to update or obtain estimated arrival time and / or destination location.

[0108] According to one embodiment, such as Figure 10The autonomous driving system 1000 shown can be comprised of the vehicle's electronic devices 200. According to one embodiment, when a user triggers an autonomous driving deactivation event during autonomous driving, the AI ​​processor 1009 of the autonomous driving system 1000 can control the vehicle's autonomous driving software to learn by inputting information related to the autonomous driving deactivation event into the training set data of a deep learning network.

[0109] Figure 11 and Figure 12 This is a block diagram example illustrating an autonomous driving mobile body according to one embodiment. Figure 13 This illustrates examples of gateways associated with user devices according to various embodiments.

[0110] Reference Figure 11 The autonomous driving mobile body 1100 according to this embodiment may include a control device 1200, perception modules 1104a, 1104b, 1104c, 1104d, an engine 1106, and a user interface 1108.

[0111] The autonomous vehicle 1100 can have an autonomous driving mode or a manual mode. For example, it can switch from manual mode to autonomous driving mode or vice versa based on user input received through the user interface 1108.

[0112] When the mobile body 1100 is running in automatic driving mode, the automatic driving mobile body 1100 can operate under the control of the control device 1200.

[0113] In this embodiment, the control device 1200 may include a controller 1220 having a memory 1222 and a processor 1224, a sensor 1210, a communication device 1230, and an object detection device 1240.

[0114] The object detection device 1240 can perform all or part of the functions of the distance measuring device.

[0115] In other words, in this embodiment, the object detection device 1240 is a device for detecting objects located outside the moving body 1100. The object detection device 1240 can detect objects located outside the moving body 1100 and generate object information based on the detection results.

[0116] Object information can include information about whether an object exists, the object's location, the distance between the moving object and the moving body, and the relative speed between the moving body and the object.

[0117] The objects can include various objects located outside the moving body 1100, such as lane lines, other vehicles, pedestrians, traffic signals, light, roads, structures, speed bumps, terrain features, and animals. Traffic signals can include traffic lights, traffic signs, and patterns or text painted on the road surface. Furthermore, light can be light produced by lights on other vehicles, light from streetlights, or sunlight.

[0118] In addition, structures can be objects located around the road and fixed to the ground. For example, structures can include streetlights, roadside trees, buildings, utility poles, traffic lights, and bridges. Topographical features can include mountains, hills, etc.

[0119] The object detection device 1240 described above may include a camera module. The controller 1220 can extract object information from external images captured by the camera module and process related information.

[0120] In addition, the object detection device 1240 may also include imaging equipment for sensing the external environment. Besides LIDAR, RADAR, GPS devices, odometry and other computer vision devices, ultrasonic sensors, and infrared sensors can also be used. These devices can be operated selectively or simultaneously as needed to achieve more accurate detection.

[0121] On the other hand, according to one embodiment of the present invention, the distance measuring device can calculate the distance between the autonomous driving mobile body 1100 and the object, and in conjunction with the control device 1200 of the autonomous driving mobile body 1100, control the movement of the mobile body based on the calculated distance.

[0122] For example, when there is a possibility of a distance conflict between the autonomous mobile vehicle 1100 and an object, the autonomous mobile vehicle 1100 can reduce its speed or stop by controlling the brakes. Another example is that when the object is a moving object, the autonomous mobile vehicle 1100 can maintain a predetermined distance from the object by controlling its driving speed.

[0123] According to one embodiment of the present invention, this distance measuring device can be configured as a module in the control device 1200 of the autonomous driving mobile body 1100. That is, the memory 1222 and processor 1224 of the control device 1200 can implement the anti-collision method described in the present invention in software.

[0124] Furthermore, sensor 1210 can be connected to sensing modules 1104a, 1104b, 1104c, and 1104d to acquire various sensing information about the internal / external environment of the moving body. Sensor 1210 may include attitude sensors (e.g., yaw, roll, and pitch sensors), collision sensors, wheel sensors, speed sensors, tilt sensors, weight sensors, heading sensors, gyro sensors, position modules, forward / backward sensors, battery sensors, fuel sensors, tire sensors, steering sensors detected by steering wheel rotation, internal temperature sensors, internal humidity sensors, ultrasonic sensors, light sensors, accelerator pedal position sensors, brake pedal position sensors, etc.

[0125] Therefore, sensor 1210 can acquire sensing signals about the following: mobile body attitude information, mobile body collision information, mobile body direction information, mobile body position information (GPS information), mobile body angle information, mobile body speed information, mobile body acceleration information, mobile body tilt information, mobile body forward / backward information, battery information, fuel information, tire information, mobile body lighting information, mobile body internal temperature information, mobile body internal humidity information, steering wheel rotation angle, mobile body external illumination, pressure on the accelerator pedal, and pressure on the brake pedal, etc.

[0126] In addition, sensor 1210 may also include other sensors, such as accelerator pedal sensor, pressure sensor, engine speed sensor, air flow sensor (AFS), intake air temperature sensor (ATS), coolant temperature sensor (WTS), throttle position sensor (TPS), TDC sensor, and crankshaft angle sensor (CAS).

[0127] As described above, sensor 1210 can generate motion state information based on sensing data.

[0128] Wireless communication device 1230 is configured to enable wireless communication between autonomous vehicles 1100. For example, autonomous vehicle 1100 can communicate with a user's mobile phone, other wireless communication devices 1230, other vehicles, central devices (such as traffic control devices), servers, etc. Wireless communication device 1230 can send and receive wireless signals according to an access wireless protocol. Wireless communication protocols may include, but are not limited to, Wi-Fi, Bluetooth, Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), and Global Systems for Mobile Communications (GSM).

[0129] Furthermore, according to this embodiment, the autonomous driving mobile unit 1100 can also achieve communication between mobile units via the wireless communication device 1230. That is, the wireless communication device 1230 can communicate with other mobile units and other vehicles on the road via vehicle-to-vehicle (V2V) communication. The autonomous driving mobile unit 1100 can send and receive data such as driving warnings and traffic information via vehicle-to-vehicle communication, and can also request information from or receive requests from other mobile units. For example, the wireless communication device 1230 can achieve V2V communication via a dedicated short-range communication (DSRC) device or a cellular-to-V2V (C-V2V) device. In addition to vehicle-to-vehicle communication, the wireless communication device 1230 can also achieve vehicle-to-everything (V2X) communication between vehicles and other things (such as electronic devices carried by pedestrians).

[0130] In addition, the wireless communication device 1230 can also obtain information from various mobility entities, such as infrastructure located on the road (e.g., traffic lights, closed-circuit television, RSU, eNode B, etc.) or other autonomous / non-autonomous driving vehicles, through a non-terrestrial network (not a terrestrial network), and use it as the information required for the autonomous driving mobility entity 1100 to perform autonomous driving.

[0131] For example, the wireless communication device 1230 can communicate wirelessly with low Earth orbit (LEO), medium Earth orbit (MEO), geostationary orbit (GEO), and high altitude platform (HAP) systems that make up the non-terrestrial network via a dedicated antenna mounted on the autonomous vehicle 1100.

[0132] For example, the wireless communication device 1230 can communicate wirelessly with various platforms constituting the NTN according to the wireless access specifications of the 5G NR NTN (5th Generation New Radio Non-Terrestrial Network) standard specification currently under discussion by organizations such as 3GPP, but is not limited thereto.

[0133] In this embodiment, the controller 1220 can consider various information such as the location, current time, and available power of the autonomous driving mobile body 1100, select a platform that can appropriately perform NTN communication, and control the wireless communication device 1230 to perform wireless communication with the selected platform.

[0134] In this embodiment, the controller 1220, as a unit controlling the overall operation of the various units within the mobile body 1100, can be configured by the manufacturer of the mobile body during manufacturing, or further configured after manufacturing to perform autonomous driving functions. Alternatively, it may include a configuration that continuously performs additional functions by upgrading the controller 1220 configured during manufacturing. Such a controller 1220 may also be named an ECU (Electronic Control Unit).

[0135] The controller 1220 can collect various data from connected sensors 1210, object detection device 1240, communication device 1230, etc., and based on the collected data, transmit control signals to other components within the mobile body, including sensors 1210, engine 1106, user interface 1108, communication device 1230, and object detection device 1240. Furthermore, although not specifically described, the control signals can also be transmitted to acceleration devices, braking systems, steering devices, or navigation devices related to the movement of the mobile body.

[0136] In this embodiment, the controller 1220 can control the engine 1106. For example, when the autonomous vehicle 1100 detects a speed limit on the road, the controller 1220 can control the engine 1106 to ensure that the driving speed does not exceed the speed limit, or to accelerate the driving speed of the autonomous vehicle 1100 within the speed limit range.

[0137] Furthermore, during the operation of the autonomous vehicle 1100, when the autonomous vehicle 1100 approaches or deviates from the lane lines, the controller 1220 determines whether this approach or deviation is a normal driving situation or another driving situation, and controls the engine 1106 to adjust the vehicle's driving based on the determination result. Specifically, the autonomous vehicle 1100 can detect the lane lines formed on both sides of the vehicle's driving lane. In this case, the controller 1220 can determine whether the autonomous vehicle 1100 is approaching or deviating from the lane lines, and if it determines that the autonomous vehicle 1100 is approaching or deviating from the lane lines, it can further determine whether this driving is due to a normal driving situation or another driving situation. Here, as an example of a normal driving situation, the vehicle may need to change lanes. And, as an example of another driving situation, the vehicle may not need to change lanes. If the controller 1220 determines that the autonomous vehicle 1100 approaches or deviates from the lane lines when the vehicle does not need to change lanes, it can control the autonomous vehicle 1100 to drive normally on the relevant vehicle without deviating from the lane lines.

[0138] When other moving objects or obstacles are in front of the moving body, the engine 1106 or braking system can be controlled to slow down the moving body. In addition to speed, the trajectory, driving path and steering angle can also be controlled. Alternatively, the controller 1220 can generate the required control signals to control the movement of the moving body based on the identification information of other external environments such as the moving body's driving lane lines and driving signals.

[0139] In addition to generating its own control signals, the controller 1220 can also communicate with surrounding mobile bodies or a central server, and send commands to control surrounding equipment through the received information, thereby controlling the movement of the mobile body.

[0140] Furthermore, when the position or viewing angle of the camera module 1250 changes, the controller 1220 may have difficulty accurately identifying moving objects or lane lines as in this embodiment. Therefore, to prevent this, the controller 1220 can also generate a control signal to perform calibration of the camera module 1250. Thus, in this embodiment, the controller 1220 sends a calibration control signal to the camera module 1250, ensuring that even if the installation position of the camera module changes due to vibrations or impacts caused by the movement of the autonomous vehicle 1100, it can maintain the normal installation position, orientation, and viewing angle of the camera module. The controller 1220 can generate a control signal to perform calibration of the camera module 1250 when a change exceeding a critical value occurs between the pre-stored initial installation position, orientation, and viewing angle information of the camera module 1250 and the initial installation position, orientation, and viewing angle information of the camera module 1250 measured during the driving of the autonomous vehicle 1100.

[0141] In this embodiment, the controller 1220 may include a memory 1222 and a processor 1224. The processor 1224 may execute software stored in the memory 1222 according to control signals from the controller 1220. Specifically, the controller 1220 stores data and commands required to execute the lane detection method described herein in the memory 1222, and these commands may be executed by the processor 1224 to implement one or more methods disclosed herein.

[0142] At this time, memory 1222 can be stored on a non-volatile recording medium executable by processor 1224. Memory 1222 can store software and data through appropriate internal and external devices. Memory 1222 can be composed of RAM (random access memory), ROM (read only memory), a hard disk, or a memory device connected to a dongle.

[0143] The memory 1222 can store at least the operating system (OS), user applications, and executable commands. The memory 1222 can also store application data and array data structures.

[0144] Processor 1224 may be a microprocessor or a suitable electronic processor, and may be a controller, microcontroller or state machine.

[0145] The processor 1224 can be implemented by a combination of computing devices, which may consist of a digital signal processor, a microprocessor, or a suitable combination thereof.

[0146] On the other hand, the autonomous driving mobile unit 1100 may also include a user interface 1108 for receiving user input to the control device 1200. The user interface 1108 allows the user to input information through an appropriate interactive method. For example, it can be implemented through a touchscreen, keyboard, operation buttons, etc. The user interface 1108 sends the input or commands to the controller 1220, and the controller 1220 executes control actions of the mobile unit based on the response to the input or commands.

[0147] Furthermore, the user interface 1108 can also communicate with devices outside the autonomous driving mobile vehicle 1100 via the wireless communication device 1230. For example, the user interface 1108 can be linked with a mobile phone, tablet computer, or other computer device.

[0148] Furthermore, while the autonomous mobile body 1100 described in this embodiment includes an engine 1106, it may also include other types of propulsion systems. For example, the mobile body may operate by electric power, or by a hybrid system of hydrogen power or a combination thereof. Therefore, the controller 1220 includes a propulsion mechanism depending on the propulsion system of the autonomous mobile body 1100, and may provide corresponding control signals to the components of each propulsion mechanism.

[0149] The following is for reference Figure 12 The detailed configuration of the control device 1100 according to this embodiment will be further described in detail.

[0150] The control device 1200 includes a processor 1224. The processor 1224 can be a general-purpose single-chip or multi-chip microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, etc. The processor can also be referred to as a central processing unit (CPU). Furthermore, in this embodiment, the processor 1224 can also be used as a combination of multiple processors.

[0151] The control device 1200 also includes a memory 1222. The memory 1222 can be any electronic component capable of storing electronic information. In addition to being a single memory, the memory 1222 can also include a combination of multiple memories 1222.

[0152] The data and command 1222a required by the distance measuring device according to the present invention to perform the distance measuring method can be stored in the memory 1222. When the processor 1224 executes the command 1222a, all or part of the command 1222a and the data 1222b required to execute the command can be loaded onto the processor 1224 (1224a, 924b).

[0153] The control device 1200 may include a transmitter 1230a, a receiver 1230b, or a transceiver 1230c for enabling signal transmission and reception. One or more antennas 1222a, 1222b may be electrically connected to the transmitter 1230a, the receiver 1230b, or each transceiver 1230c, and may also include additional antennas.

[0154] The control device 1200 may also include a digital signal processor (DSP) 1270. The DSP 1270 enables the moving body to process digital signals quickly.

[0155] The control device 1200 may also include a communication interface 1280. The communication interface 1280 may include one or more ports and / or communication modules for connecting other devices to the control device 1200. The communication interface 1280 allows a user to interact with the control device 1200.

[0156] Various components of the control device 1200 can be connected via one or more buses 1290, and the buses 1290 may include power buses, control signal buses, status signal buses, data buses, etc. Under the control of the processor 1224, the components can exchange information with each other and perform predetermined functions via the buses 1290.

[0157] On the other hand, in various embodiments, the control device 1200 can be associated with a gateway to communicate with a secure cloud. For example, see reference... Figure 12 The control device 1200 may be associated with a gateway 1305 for providing information acquired from at least one of the components (1301 to 1304) of the vehicle 1300 to the security cloud 1306. For example, the gateway 1305 may be included within the control device 1200. Alternatively, the gateway 1305 may be configured as a separate device within the vehicle 1300, distinct from the control device 1200. The gateway 1305 is capable of communicating with the software management cloud 1309, the security cloud 1306, and the internal network of the vehicle 1300 protected by the in-vehicle security software 1310, which have different networks.

[0158] For example, component 1301 may be a sensor. For example, the sensor may be used to obtain information about at least one of the state of vehicle 1300 or the state of the surroundings of vehicle 1300. For example, component 1301 may include sensor 1210.

[0159] For example, component 1302 may be an ECU (electronic control unit). For example, the ECU may be used for engine control, transmission control, airbag control, and tire pressure management.

[0160] For example, component 1303 may be an instrument cluster. For example, the instrument cluster may be a panel located in front of the driver's seat in a dashboard. For example, the instrument cluster may be configured to display information needed for driving to the driver (or passenger). For example, the instrument cluster may display at least one of the following: visual elements indicating engine revolutions per minute (RPM), visual elements indicating vehicle speed 1300, visual elements indicating remaining fuel, visual elements indicating gear status, and visual elements indicating information obtained through component 1301.

[0161] For example, component 1304 may be a telematics device. For example, the telematics device may be a device that provides various mobile communication services such as location information and safe driving within the vehicle 1300 by combining wireless communication technology and GPS (global positioning system) technology. For example, the telematics device may be used to connect the vehicle 1300 to the driver, the cloud (e.g., a secure cloud 1306), and / or the surrounding environment. For example, the telematics device may support high bandwidth and low latency for 5G NR standard technologies (e.g., 5G NR V2X technology, 5G NR NTN (Non-Terrestrial Network) technology). For example, the telematics device may support autonomous driving of the vehicle 1300.

[0162] For example, gateway 1305 can be used to connect the in-vehicle network of vehicle 1300 to software management cloud 1309 and security cloud 1306, which are external networks of vehicle 1300. For example, software management cloud 1309 can be used to update or manage at least one software required for driving and managing vehicle 1300. For example, software management cloud 1309 can be linked with in-car security software 1310 installed in the vehicle. For example, in-car security software 1310 can be used to provide security functions within vehicle 1300. For example, in-car security software 1310, in order to encrypt the in-vehicle network, can encrypt data sent and received through the in-vehicle network using an encryption key obtained from an external authorized server. In various embodiments, the encryption key used by in-car security software 1310 can be generated in response to vehicle identification information (vehicle license plate, vehicle identification number (VIN)) or information uniquely assigned to each user (e.g., user identification information).

[0163] In various embodiments, gateway 1305 can send data encrypted by vehicle security software 1310 based on the encryption key to software management cloud 1309 and / or security cloud 1306. Software management cloud 1309 and / or security cloud 1306 decrypt the data encrypted by vehicle security software 1310 using the encryption key, thereby identifying from which vehicle or user the data was received. For example, since the decryption key is a unique key corresponding to the encryption key, software management cloud 1309 and / or security cloud 1306 can identify the sender of the data (e.g., the vehicle or the user) based on the data decrypted using the decryption key.

[0164] For example, gateway 1305 is configured to support in-vehicle security software 1310 and can be associated with control unit 1200. For instance, gateway 1305 can be associated with control unit 1200 to support connectivity between control unit 1200 and client devices connected to security cloud 1306. According to another example, gateway 1305 can be associated with control unit 1200 to support connectivity between control unit 1200 and a third-party cloud 1308 connected to security cloud 1306. However, it is not limited to these examples.

[0165] In various embodiments, gateway 1305 can be used to connect vehicle 1300 to software management cloud 1309 for managing the operating software of vehicle 1300. For example, software management cloud 1309 monitors whether the operating software of vehicle 1300 needs to be updated, and based on the detection that the operating software of vehicle 1300 needs to be updated, can provide data for updating the operating software of vehicle 1300 through gateway 1305. According to another example, software management cloud 1309 receives user requests for updating the operating software of vehicle 1300 from vehicle 1300 through gateway 1305, and can provide data for updating the operating software of vehicle 1300 based on the received request. However, it is not limited to this.

[0166] Figure 14 This is a schematic diagram illustrating the operation of an electronic device for training a neural network based on a training dataset, according to one embodiment.

[0167] Reference Figure 14 The described operation can be performed by the aforementioned electronic device (e.g.: Figure 2 The electronic device 200 in the middle is used for execution.

[0168] refer to Figure 14In operation 1410, according to one embodiment, the electronic device can acquire a training dataset. The electronic device can acquire a training dataset for supervised learning. The training data may include data pairs of input data and ground truth data corresponding to the input data. The ground truth data may represent output data to be obtained from a neural network that receives input data of the ground truth data pairs. The ground truth data may be acquired by the aforementioned electronic device.

[0169] For example, when training a neural network to recognize an image, the training data may include the image and information about one or more objects contained within it. The information may include the category or class of the objects identifiable by the image. The information may include the position, width, height, and / or size of the visual object corresponding to the object in the image. The training dataset identified by operation 1410 may include multiple training data pairs. In the example described for training a neural network to recognize an image, the training dataset identified by the electronic device may include multiple images and ground truth data corresponding to each of the multiple images.

[0170] Reference Figure 14 In operation 1420, according to one embodiment, the electronic device can train a neural network based on a training dataset. In one embodiment of training the neural network based on supervised learning, the electronic device can input input data contained in the training data into the input layer of the neural network. (See also...) Figure 10 This describes an example of a neural network including the input layer. From the output layer of the neural network that receives the input data through the input layer, an electronic device can acquire the output data of the neural network corresponding to the input data.

[0171] In one embodiment, training of operation 1420 can be performed based on the difference between the output data and ground truth data contained in the training data corresponding to the input data. For example, the electronic device can adjust more than one parameter associated with the neural network (e.g., referring to a gradient descent algorithm) based on a gradient descent algorithm. Figure 10 The weights described later are used to reduce the difference. The operation of an electronic device that adjusts one or more parameters can be referred to as tuning (optimizing) a neural network. The electronic device can perform neural network tuning based on output data by using a function defined for evaluating neural network performance, such as a cost function. The difference between the output data and the ground truth data can be included in one example of the cost function.

[0172] refer to Figure 14 In operation 1430, according to one embodiment, the electronic device can identify whether the neural network trained by operation 1420 outputs valid output data. Valid output data means that the difference (or cost function) between the output data and the ground truth data satisfies the conditions set for using the neural network. For example, the electronic device can determine that the neural network has output valid output data when the average and / or maximum value of the difference between the output data and the ground truth data is less than or equal to a specified threshold.

[0173] If the neural network does not output valid output data (1430 - No), the electronic device can repeatedly perform training of the neural network based on operation 1420. The embodiment is not limited thereto; the electronic device can repeatedly perform operations 1420 and 1430.

[0174] In a state where valid output data is obtained from the neural network (1430-Yes), based on operation 1440, an electronic device according to one embodiment can use the trained neural network. For example, the electronic device can input other input data, different from the input data input to the neural network as training data, into the neural network. The electronic device can use the output data obtained from the neural network that receives the other input data as the result of inference based on the other input data by the neural network.

[0175] Figure 15 This is a block diagram of an electronic device according to one embodiment.

[0176] Figure 15 The electronic device 1501 may include the aforementioned electronic device (e.g., Figure 2 Electronic devices 200).

[0177] For example, refer to Figure 14 The instructions can be provided by Figure 15 Electronic device 1501 and / or Figure 15 The processor 1510 executes.

[0178] Reference Figure 15The processor 1510 of the electronic device 1501 can perform computations related to the neural network 1530 stored in the memory 1520. The processor 1510 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU). The NPU can be implemented as a separate chip from the CPU, or integrated on a chip such as a CPU as a system-on-a-chip (SoC). An NPU integrated into a CPU can be referred to as a neural core and / or an artificial intelligence (AI) accelerator.

[0179] Reference Figure 15 The processor 1510 can recognize the neural network 1530 stored in the memory 1520. The neural network 1530 may include a combination of an input layer 1532, one or more hidden layers 1534 (or intermediate layers) and an output layer 1536. Each of the aforementioned layers (e.g., input layer 1532, one or more hidden layers 1534, and output layer 1536) may include multiple nodes. The number of hidden layers 1534 may vary depending on the embodiment, and a neural network 1530 containing multiple hidden layers 1534 may be referred to as a deep neural network. The operation of training the deep neural network may be referred to as deep learning.

[0180] In one embodiment, if the neural network 1530 has a feedforward neural network structure, a first node contained in a particular layer can be connected to all second nodes contained in other layers preceding that particular layer. In memory 1520, parameters stored for the neural network 1530 may include weights assigned to the connections between the second and first nodes. In the neural network 1530 with a feedforward neural network structure, the value of the first node may correspond to a weighted sum of the values ​​assigned to the second node based on the weights assigned to the connections connecting the second and first nodes.

[0181] In one embodiment, if the neural network 1530 has a convolutional neural network structure, a first node contained in a particular layer may correspond to a weighted sum of a subset of second nodes contained in other layers preceding that particular layer. A subset of the second nodes corresponding to the first node may be identified by a filter corresponding to the particular layer. Parameters stored for the neural network 1530 in memory 1520 may include weights representing the filters. The filters may include one or more nodes in the second nodes used to compute the weighted sum of the first nodes, and weights corresponding to each of the one or more nodes.

[0182] According to one embodiment, the processor 1510 of the electronic device 1501 can train the neural network 1530 using a training dataset 1540 stored in the memory 1520. Based on the training dataset 1540, the processor 1510 can perform reference... Figure 14 The described operation adjusts the neural network 1530 to store one or more parameters in memory 1520.

[0183] According to one embodiment, the processor 1510 of the electronic device 1501 can perform object detection, object recognition, and / or object classification using a neural network 1530 trained on a training dataset 1540. The processor 1510 can input images (or videos) acquired by a camera 1550 into the input layer 1532 of the neural network 1530. Based on the input layer 1532 where the images are input, the processor 1510 can sequentially obtain the values ​​of each node in the neural network 1530, thereby obtaining a set of values ​​for the nodes of the output layer 1536 (e.g., output data). The output data can be used as the result of inference using the neural network 1530 on information contained in the images. The embodiment is not limited thereto; the processor 1510 can also acquire images (or videos) from an external electronic device connected to the electronic device 1501 via a communication circuit 1560 and input them into the neural network 1530.

[0184] In one embodiment, a neural network 1530 trained for image processing can be used to identify regions in the image corresponding to objects (object detection), and / or to identify the categories of objects represented in the image (object recognition and / or object classification). For example, electronic device 1501 can utilize neural network 1530 to segment regions in the image corresponding to the objects based on rectangular shapes such as bounding boxes. For example, electronic device 1501 can utilize neural network 1530 to identify at least one category matching the object from a plurality of specified categories.

[0185] As described above, the electronic device 200 may include a camera 230. The electronic device 200 may include a memory 220. The electronic device 200 may include a processor 210. The processor 210 may be configured to: acquire an image 511 using the camera 230. The processor 210 may be configured to: acquire spatial information about the external space at the time the image 511 is acquired using the image 511. The processor 210 may be configured to: acquire state information about a target object contained in the external space using the spatial information. The processor 210 may be configured to: execute a first language model 541 using the state information to acquire first text describing the motion of the target object. The processor 210 may be configured to: execute a second language model 542 using the first text and the image 511 to acquire second text describing the image 511.

[0186] The processor 210 can be configured to generate driving information for controlling the vehicle 310 using the second text.

[0187] The processor 210 can be configured to use the second text to generate guidance information for informing the user of an anomaly caused by the target object.

[0188] The electronic device 200 may include a speaker 240. The processor 210 may be configured to control the speaker 240 using the boot information.

[0189] The processor 210 can be configured to: identify features of the target object using the spatial information. The processor 210 can also be configured to: acquire the state information of the target object based on the features.

[0190] The features may include: the shape of the target object and the speed of the target object.

[0191] The status information may include at least one of the following: the position of the target object, the movement trajectory of the target object, the action of the target object, and the size of the target object.

[0192] The processor 210 may be configured to: execute the first language model 541 using the second text to obtain a third text for reducing the error between the first text and the second text. The processor 210 may also be configured to: execute the second language model 542 using the third text and the image 511 to obtain a fourth text for describing the image 511.

[0193] The electronic device 200 may include at least one sensor 250. The processor 210 may be configured to acquire sensing data 513 using the at least one sensor 250. The processor 210 may be configured to acquire the spatial information using the image 511 and the sensing data 513.

[0194] The processor 210 can be configured to: recognize user input. The processor 210 can be configured to: generate information for controlling the vehicle 310 using the user input and the second text.

[0195] The electronic device 200 may include a microphone 251. The processor 210 may be configured to receive the user input via the microphone 251.

[0196] The first language model 541 may include a physics-informed model for predicting the motion of the target object from the state information about the target object.

[0197] As described above, the method performed by the electronic device 200 including the camera 230 may include the following operations: acquiring an image 511 using the camera 230. The method may include the following operations: acquiring spatial information about the external space at the time the image 511 was acquired using the image 511. The method may include the following operations: acquiring state information about a target object contained in the external space using the spatial information. The method may include the following operations: executing a first language model 541 using the state information to obtain first text describing the motion of the target object. The method may include the following operations: executing a second language model 542 using the first text and the image 511 to obtain second text describing the image 511.

[0198] The method may include the following operation: using the second text to generate driving information for controlling vehicle 310.

[0199] The method may include the following operation: using the second text to generate guidance information for informing the user of the danger posed by the target object.

[0200] The method may include the following operations: using the spatial information to identify the features of the target object. The method may also include the following operations: based on the features, obtaining the state information of the target object.

[0201] The method may include the following operations: executing the first language model 541 using the second text to obtain a third text for reducing the error between the first text and the second text. The method may also include the following operations: executing the second language model 542 using the third text and the image 511 to obtain a fourth text for describing the image 511.

[0202] The electronic device 200 may include a sensor. The method may include the following operations: acquiring sensing data 513 using the sensor. The method may also include the following operations: acquiring the spatial information using the image 511 and the sensing data 513.

[0203] The method may include the following operations: recognizing user input. The method may include the following operations: using the user input and the second text, generating information for controlling the vehicle (310).

[0204] As described above, a non-transitory computer-readable storage medium is used to store one or more programs, wherein, when executed by a processor 210 of an electronic device 200 including a camera 230, the electronic device 200 is capable of: acquiring an image 511 using the camera 230; acquiring spatial information about the external space at the time the image 511 is acquired using the image 511; acquiring state information about a target object contained in the external space using the spatial information; executing a first language model 541 using the state information to acquire first text describing the movement of the target object; and executing a second language model 542 using the first text and the image 511 to acquire second text describing the image 511.

Claims

1. An electronic device (200), comprising: Camera (230); Memory (220); as well as Processor (210) The processor (210) is configured as follows: Image (511) is acquired using the camera (230); Spatial information of the external space at the time the image (511) is acquired is obtained using the image (511); The spatial information is used to obtain state information about the target object contained in the external space; Utilizing the state information, a first language model (541) is executed to obtain first text describing the motion of the target object; and A second language model (542) is executed using the first text and the image (511) to obtain a second text for describing the image (511).

2. The electronic device (200) according to claim 1, characterized in that, The processor (210) is configured to: Using the second text, driving information for controlling the vehicle (310) is generated.

3. The electronic device (200) according to claim 1, characterized in that, The processor (210) is configured to: Using the second text, guide information is generated to inform the user of the anomaly caused by the target object.

4. The electronic device (200) according to claim 3, characterized in that, Includes speakers (240). The processor (210) is configured as follows: The speaker (240) is controlled using the guidance information.

5. The electronic device (200) according to claim 1, characterized in that, The processor (210) is configured to: Using the spatial information, the characteristics of the target object are identified; and Based on the aforementioned features, the state information of the target object is obtained.

6. The electronic device (200) according to claim 5, characterized in that, The features include: The shape of the target object and the speed of the target object.

7. The electronic device (200) according to claim 5, characterized in that, The status information includes: At least one of the following: the position of the target object, the trajectory of the target object, the action of the target object, and the size of the target object.

8. The electronic device (200) according to claim 1, characterized in that, The processor (210) is configured to: The first language model (541) is executed using the second text to obtain a third text for reducing the error between the first text and the second text; and The second language model (542) is executed using the third text and the image (511) to obtain a fourth text for describing the image (511).

9. The electronic device (200) according to claim 1, characterized in that, Includes at least one sensor (250). The processor (210) is configured as follows: Sensing data (513) is acquired using the at least one sensor (250); and The spatial information is obtained using the image (511) and the sensing data (513).

10. The electronic device (200) according to claim 1, characterized in that, The processor (210) is configured to: Recognize user input; and Using the user input and the second text, information for controlling the vehicle (310) is generated.

11. The electronic device (200) according to claim 10, characterized in that, Includes microphone (251), The processor (210) is configured as follows: The user input is received via the microphone (251).

12. The electronic device (200) according to claim 1, characterized in that, The first language model (541) includes a physics-informed model, which is used to predict the motion of the target object from the state information of the target object.

13. A method performed by an electronic device (200), the electronic device (200) including a camera (230), wherein the method includes the following operations: Image (511) is acquired using the camera (230); Spatial information of the external space at the time the image (511) is acquired is obtained using the image (511); The spatial information is used to obtain state information about the target object contained in the external space; The first language model (541) is executed using the state information to obtain first text describing the motion of the target object; as well as A second language model (542) is executed using the first text and the image (511) to obtain a second text for describing the image (511).

14. The method according to claim 13, characterized in that, This includes the following operations: Using the second text, driving information for controlling the vehicle (310) is generated.

15. The method according to claim 13, characterized in that, This includes the following operations: Using the second text, guide information is generated to inform the user of the dangers posed by the target object.

16. The method according to claim 13, characterized in that, This includes the following operations: Using the spatial information, the characteristics of the target object are identified; and Based on the aforementioned features, the state information of the target object is obtained.

17. The method according to claim 13, characterized in that, This includes the following operations: The first language model (541) is executed using the second text to obtain a third text for reducing the error between the first text and the second text; and The second language model (542) is executed using the third text and the image (511) to obtain a fourth text for describing the image (511).

18. The method according to claim 13, characterized in that, The electronic device (200) includes at least one sensor (250). The method includes the following operations: Sensing data (513) is acquired using the at least one sensor (250); and The spatial information is obtained using the image (511) and the sensing data (513).

19. The method according to claim 13, characterized in that, This includes the following operations: Recognize user input; and Using the user input and the second text, information for controlling the vehicle (310) is generated.

20. A non-transitory computer-readable storage medium for storing one or more programs, which, when executed by a processor (210) of an electronic device (200) including a camera (230), enable the electronic device (200) to: Image (511) is acquired using the camera (230); Spatial information of the external space at the time the image (511) is acquired is obtained using the image (511); The spatial information is used to obtain state information about the target object contained in the external space; The first language model (541) is executed using the state information to obtain first text describing the motion of the target object; as well as A second language model (542) is executed using the first text and the image (511) to obtain a second text for describing the image (511).

Images