Electronic device for configuring a leading vehicle in a fout of vehicles and method thereof

Abstract

The invention relates to an electronic device of a leading vehicle configured in a fleet of vehicles and a method thereof. An electronic device for fleet driving of vehicles may be configured to: acquire a front image; generating a cut image; transmitting at least one of the front image and the cropped image to each of the other electronic devices; acquiring first information related to an external object from the external object included in at least one of the front image and the cropped image; receiving, from the other electronic device, second information related to the external object acquired from the external object included in at least the other of the front image and the cut image; the external object is identified based on the first information and the second information.

Description

Technical Field

[0001] The following description relates to an electronic device and method for identifying external objects of vehicles traveling in a fleet. Background Art

[0002] Platooning is a technology for controlling autonomous driving of two or more vehicles. The vehicles that make up a platoon can travel in a certain formation. Platooning can reduce air resistance by reducing the gaps between vehicles, thereby improving fuel efficiency, reducing the risk of accidents, and alleviating traffic congestion by controlling vehicle flow. The vehicles that make up a platoon may include a leading vehicle and following vehicles. Electronic devices configured in the leading vehicle can control platooning. For example, the electronic device can identify external objects located in front of the leading vehicle, set a driving route based on the external objects, and control the speed and direction of the vehicle. Summary of the Invention

[0003] Problems to be solved by the invention

[0004] The electronic device installed in the lead vehicle can identify external objects from images captured by the camera. Research is underway to reduce the data processing burden on the electronic device.

[0005] Means used to solve problems

[0006] An electronic device configured for a leading vehicle in a platooning vehicle is provided. The electronic device may include: a processor, a memory for storing instructions, and a camera. When executed by the processor, the instructions may cause the electronic device to perform the following actions: using the camera to acquire a front image in front of the leading vehicle, generating a cropped image by cropping the front image, sending at least one of the front image and the cropped image to each other electronic device configured in each following vehicle, acquiring first information related to an external object from the external object included in at least one of the front image and the cropped image, receiving second information related to the external object acquired from the external object included in at least the other of the front image and the cropped image from the other electronic device, and identifying the external object based on the first information and the second information.

[0007] An electronic device configured for a leading vehicle in a convoy of vehicles is provided. The electronic device may include: a processor, a memory for storing instructions, and a camera. When executed by the processor, the instructions may cause the electronic device to perform the following actions: using each of the cameras to acquire a front image in front of the leading vehicle, transmitting the front image to each other electronic device configured in each following vehicle, acquiring first information related to an external object included in at least one of the front images, receiving second information related to the external object acquired from the external object included in at least one of the front images from the other electronic devices, and identifying the external object based on the first and second information.

[0008] A method is provided for an electronic device configured in a leading vehicle in a convoy of vehicles. The method may include the following steps: using a camera of the electronic device to acquire a front image in front of the leading vehicle, cropping the front image to generate a cropped image, transmitting at least one of the front image and the cropped image to each other electronic device configured in each following vehicle, acquiring first information related to an external object from the external object included in at least one of the front image and the cropped image, receiving second information related to the external object acquired from the external object included in at least the other of the front image and the cropped image from the other electronic device, and identifying the external object based on the first and second information.

[0009] Effects of the Invention

[0010] When controlling the vehicles in the convoy driving mode, the electronic device recognizes external objects using information acquired by the electronic device and information provided from other electronic devices, so that the external objects can be recognized quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Schematically illustrating vehicles traveling in a convoy.

[0012] Figure 2 is a block diagram of an electronic device for platooning of vehicles according to an embodiment.

[0013] Figure 3A The field of view according to the viewing angle of the camera is schematically shown.

[0014] Figure 3B Shows images according to viewing angles.

[0015] Figure 4 is a flowchart illustrating the operation of an electronic device for recognizing an external object.

[0016] Figure 5 Shows vehicles traveling in a convoy.

[0017] Figure 6A 、 Figure 6B and Figure 6C Shows the front image and the cropped image.

[0018] Figure 7A is a flowchart illustrating an operation of an electronic device for recognizing an external object by transmitting a cropped image to another electronic device.

[0019] Figure 7B This is a flowchart illustrating the operation of an electronic device that recognizes external objects through distributed computing or edge computing.

[0020] Figure 7C The present invention is a signal flow diagram for explaining the steps of an electronic device of a vehicle discovering other electronic devices of other vehicles for requesting work sharing according to a distributed computing method.

[0021] Figure 8 FIG. 1 is a flowchart illustrating an operation of an electronic device recognizing an external object based on a main image and a sub-image according to an embodiment.

[0022] Figure 9 Schematically showing the Figure 8 The actions shown in FIG. 1 are the process of processing the main image and the sub-image.

[0023] Figure 10 This is a flowchart illustrating the operation of an electronic device using multiple cameras.

[0024] Figure 11 FIG. 1 is a flowchart illustrating an operation of an electronic device recognizing an external object based on a main image and a sub-image according to an embodiment.

[0025] Figure 12 An example of a block diagram of an automatic driving system for a vehicle according to an embodiment is shown.

[0026] Figure 13 and Figure 14 An example of a block diagram of an autonomous driving mobile body according to an embodiment is shown.

[0027] Figure 15 An example of a gateway associated with a user device according to various embodiments is shown.

[0028] Figure 16 is a diagram for explaining the operation of an electronic device for training a neural network based on a learning data set according to one embodiment.

[0029] Figure 17 is a block diagram of an electronic device according to an embodiment.

[0030] Figure 18A and Figure 18B An example of vehicles performing convoy driving is shown.

[0031] Figure 19A 、 Figure 19B 、 Figure 19C and Figure 19D An example of various scenarios of wireless resource allocation for direct communication between electronic devices according to an embodiment is shown. DETAILED DESCRIPTION

[0032] The electronic devices according to various embodiments disclosed herein may be various types of devices. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or household appliances. The electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0033] It should be understood that the various embodiments herein and the terms used therein are not intended to limit the technical features described herein to specific embodiments, but rather include various modifications, equivalents or substitutes of the embodiments. In conjunction with the description of the accompanying drawings, similar figure numerals may be used for similar or related constituent elements. Unless otherwise clearly indicated in the relevant context, the singular form of the noun corresponding to a certain project may include one or more of the above-mentioned projects. Herein, phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one of the projects listed together in the corresponding phrases or any possible combination thereof. Terms such as "first" or "second" may be simply used to distinguish one constituent element from another constituent element, and these constituent elements are not limited in other respects (such as importance or order). If a certain component (for example, a first component) is referred to as being “coupled” or “connected” to another component (for example, a second component), with or without the term “functionally” or “communicatively”, it means that the certain component can be connected to the other component directly (for example, by wire), wirelessly, or through a third component.

[0034] The term "module" used in various embodiments herein may include units implemented by hardware, software, or firmware, and may be interchangeable with terms such as logic, logic block, component, or circuit. A module may be an integral component or a minimum unit or portion thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0035] The various embodiments herein may be implemented as a storage medium (e.g., software (e.g., a program) comprising one or more instructions stored in a built-in memory or an external memory) that can be read by a machine (e.g., an electronic device 100). For example, a processor (e.g., processor 110) of a machine (e.g., an electronic device 100) may call and execute at least one of the one or more stored instructions from a storage medium. This allows the machine to be run to perform at least one function according to at least one of the called instructions. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The storage medium readable by a machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" refers only to the fact that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves). The term does not distinguish between cases where data is stored in a semi-permanent manner and cases where data is stored temporarily in a storage medium.

[0036] According to one embodiment, the methods according to various embodiments disclosed herein may be provided in a computer program product. As a commodity, the computer program product can be traded between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or through an app store (e.g., the Play Store). TM ) or distributed (e.g., downloaded or uploaded) directly or online between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, an app store's server, or a relay server.

[0037] According to various embodiments, each constituent element (for example: module or program) in the above-mentioned constituent elements can include single or multiple individuals, and some of the multiple individuals can be configured separately from other constituent elements. According to various embodiments, one or more constituent elements or actions in the aforementioned corresponding constituent elements can be omitted, or one or more other constituent elements or actions can be added. Alternatively or additionally, multiple constituent elements (for example: module or program) can be integrated into one constituent element. In this case, the integrated constituent element can perform one or more functions of each constituent element in the multiple constituent elements in the same or similar manner as the function performed by the corresponding constituent element in the multiple constituent elements before being integrated. According to various embodiments, the actions performed by modules, programs or other constituent elements can be performed sequentially, in parallel, iteratively or heuristically, or one or more actions in the actions can be performed in different orders, or omitted, or one or more other actions can be added.

[0038] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0039] Figure 1 Schematically illustrating vehicles traveling in a convoy.

[0040] Platooning is a technology that controls two or more vehicles 10 that form a platoon to travel in a specified formation. Each of the vehicles 10 may include an electronic device for platooning (e.g., Figure 2 The electronic devices 100 and 200 may utilize wireless communication technology to share control information of the vehicle 10 and information collected by the electronic devices 100 and 200 configured in each vehicle 10 in real time. Figure 1 The wireless communication technology shown for exchanging information between the electronic devices 100 and 200 can use various wireless access technologies (Wireless Access Technologies) such as Vehicle to Infrastructure (V2I), Vehicle to Device (V2D), Vehicle to Vehicle (V2V), Vehicle to Pedestrian (V2P), etc., 5G New Radio (NR) sidelink in cellular mode, and Dedicated Short Range Communication (DSRC) based on 802.11.

[0041] The vehicles 10 can be divided into a leading vehicle 11 and a following vehicle 12. The leading vehicle 11 may refer to the vehicle at the front of the convoy of vehicles 10, and the following vehicles 12 may refer to the remaining vehicles except the leading vehicle 11. The electronic device 100 configured in the leading vehicle 11 can be used to control the overall movement of the convoy. For example, since the leading vehicle 11 is at the front of the convoy, it can include more electronic components (e.g., radar, lidar, camera) than other vehicles. The electronic device 100 can obtain more diverse information than other electronic devices 200. The electronic device 100 according to one embodiment can have higher performance than other electronic devices 200.

[0042] The electronic device 100 can transmit and / or receive data with an external electronic device (e.g., a base station 13 and / or a satellite 14). For example, the electronic device 100 can receive data including information related to a driving route from the external electronic devices 13 and 14 to determine the driving route, and can transmit data including information related to the real-time location of the fleet to the external electronic devices 13 and 14.

[0043] The electronic device 100 may be configured to control the travel of the vehicles 10 in the convoy based on information related to the vehicles 10 in the convoy (e.g., the travel route, travel speed, the distance between vehicles 10, and / or the formation of the convoy) and / or information related to the surrounding environment. For example, the electronic device 100 may transmit a signal for controlling the travel of the convoy to each other electronic device 200 configured in each following vehicle 12. The other electronic devices 200 may be configured to control the travel of the following vehicles 12 based on the signal received from the electronic device 100.

[0044] Information related to the surrounding environment may include information related to external objects 30. For example, external objects 30, which are objects located around the driving route of vehicle 10, may be objects that need to be considered for safe driving. For example, external objects 30 may include other vehicles 20, lane lines 31, lanes 32, traffic lights 33, sidewalks 34, pedestrians 35, and / or speed bumps 36, but are not limited thereto.

[0045] According to an embodiment, the electronic device 100 may include a camera (eg: Figure 2 Camera 140 can capture images and / or videos of the front of the leading vehicle 11. For example, information related to the surrounding environment can be identified using the images and / or videos captured by camera 140. However, this is not a limitation. Camera 140 can capture images and / or videos of the surrounding environment of the leading vehicle 11 by capturing images from the sides and rear of the leading vehicle 11.

[0046] According to one embodiment, the electronic device 100 configured in the leading vehicle 11 is located at the front of the queue of the convoy, and therefore can be configured to provide various functions. For example, since the other electronic devices 200 configured in the following vehicles 12 follow the leading vehicle 11, it may be difficult to obtain information in front of the convoy, so the information in front can be obtained through the electronic device 100. The electronic device 100 can determine the driving route of the convoy based on the information in front, and can send a signal for controlling the driving of the following vehicle 12 so that the vehicle 10 drives based on the determined driving route. The electronic device 100 can receive information related to the surrounding environment of the following vehicle 12 from the other electronic devices 200, and control the driving of the convoy based on the information. The electronic device 100 can send and / or receive data with the external electronic devices 13 and 14.

[0047] The electronic device 100 provides a variety of functions, and therefore, the information processed by the electronic device 100 and the actions performed by the electronic device 100 are numerous. Providing the various functions of the electronic device 100 requires high-performance electronic components, which may sometimes cause errors. For example, as the electronic device 100 processes various information, there may be a delay in recognizing an external object 30 from the forward image captured by the camera 140, or there may be an error in recognizing the external object 30. The delay in recognizing the external object 30 or the error in recognizing the external object 30 may cause an accident involving the vehicle 10.

[0048] The electronic device 100 according to one embodiment can transmit the front image acquired by the camera 140 or a cropped image generated from the front image to the other electronic device 200, and use the information provided by the other electronic device 200 to recognize the external object 30. Therefore, the amount of data to be processed by the electronic device 100 for recognizing the external object 30 is reduced, and the burden of data processing can be alleviated, so that the external object 30 can be accurately recognized.

[0049] Hereinafter, an electronic device 100 according to an embodiment will be described with reference to the accompanying drawings.

[0050] Figure 2 is a block diagram of an electronic device for platooning of vehicles according to an embodiment.

[0051] Reference Figure 2 According to one embodiment, the electronic device 100 may include a processor 110, a memory 120, a wireless communication device 130, a camera 140, and / or a GPS (Global Positioning System) sensor 150. According to one embodiment, the electronic device 100 may be configured on a leading vehicle (e.g., Figure 1 electronic device in the lead vehicle 11).

[0052] For example, the processor 110, the memory 120, the wireless communication device 130, the camera 140, and / or the GPS sensor 150 may be electrically and / or operatively connected to each other via electronic components such as a communication bus. Hereinafter, "operably coupled hardware" means that a direct or indirect connection between the hardware is established in a wired or wireless manner, such that a first piece of hardware controls a second piece of hardware.

[0053] exist Figure 2 In FIG, the processor 110, the memory 120, the camera 140, the wireless communication device 130 and / or the GPS sensor 150 are shown as different plots, but are not limited thereto. Figure 2 Some of the hardware shown may be implemented as part of a single integrated circuit (IC) or a single package, such as a system on a chip (SoC).

[0054] According to one embodiment, the memory 120 may store instructions. The processor 110 may be configured to process data based on the instructions stored in the memory 120. For example, the processor 110 may include an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The processor 110 may have a single-core processor structure, or may have a multi-core processor structure such as a dual-core, quad-core, hexa-core, or octa-core processor.

[0055] According to one embodiment, the memory 120 may include a hardware component for storing data and / or instructions executable by the processor 110. For example, the memory 120 may include a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). For example, the volatile memory may include at least one of a dynamic random access memory (DRAM), a static random access memory (SRAM), a cache RAM, and a pseudo-static random access memory (PSRAM). For example, the non-volatile memory may include at least one of a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a hard disk, an optical disk, a solid state drive (SSD), and an embedded multi-media card (eMMC). For example, the memory 120 of the electronic device 100 may include an image processing model (e.g., a neural network model) 121. The electronic device 100 may recognize an external object (e.g., a location) based on the image processing model 121 (e.g., a neural network model) stored in the memory 120. Figure 1 external object 30). Hereinafter, the image processing model may be referred to as a neural network model.

[0056] According to one embodiment, the wireless communication device 130 can be used to communicate wirelessly with other electronic devices 200 and / or external electronic devices. For example, the electronic device 100 can be configured to use the wireless communication device 130 to communicate with external electronic devices (e.g., base stations (e.g., Figure 1 Base stations 13) and / or satellites (e.g. Figure 1 The wireless communication device 130 may be electrically connected to an antenna (e.g., a satellite 14) or other electronic devices 200 for transmitting and / or receiving signals. Figure 14The wireless communication device 130 can convert the analog signal provided by the processor 110 into a digital signal and upconvert the baseband signal into a radio frequency (RF) signal. The electronic device 100 can use the GPS sensor 150 to obtain information related to the real-time location of the fleet and can use the wireless communication device 130 to send data including the information to the external electronic devices 13 and 14. The electronic device 100 can use the data for controlling the following vehicle (for example: Figure 1 The signal of the following vehicle 12 is sent to the wireless communication device 230 of the other electronic device 200. The other electronic device 200 can receive the signal through the wireless communication device 230.

[0057] According to one embodiment, the camera 140 may include a lens assembly or an image sensor. The lens assembly may collect light emitted from a subject to be imaged. The lens assembly may include more than one lens. For example, the camera 140 may include multiple lens assemblies. For example, in the camera 140, some of the multiple lens assemblies may have the same lens properties (e.g., viewing angle, focal length, autofocus, focal ratio (f-number), or optical zoom), or at least one lens assembly may have one or more lens properties that differ from those of the other lens assemblies. The lens assembly may include a wide-angle lens or a telephoto lens. For example, the electronic device 100 may include a flash for the camera 140. The flash may include one or more light-emitting diodes (e.g., red-green-blue (RGB) LEDs, white LEDs, infrared LEDs, or ultraviolet LEDs) or a xenon lamp. For example, the image sensor may capture an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly into an electrical signal. According to one embodiment, the image sensor may include one image sensor selected from a plurality of image sensors having different properties (e.g., an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor), a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor may be implemented, for example, using a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0058] According to one embodiment, the electronic device 100 can use the camera 140 to identify the surrounding environment of the leading vehicle 11. For example, the electronic device 100 can identify the external object 30 based on the image captured by the camera 140. For example, the electronic device 100 can use the image processing model 121 (e.g., a neural network model) to identify the external object 30 corresponding to the image captured by the camera 140. For example, the electronic device 100 can capture an image corresponding to another vehicle 20 through the camera 140 and identify the other vehicle 20 from the image.

[0059] The other electronic devices 200 configured in the following vehicle 12 may include substantially the same components as the electronic device 100 configured in the leading vehicle 11. For example, each other electronic device 200 may include a processor 210, a memory 220, a wireless communication device 230, a camera 240, and / or a GPS sensor 250. The aforementioned description of the components of the electronic device 100 may be substantially identically applied to the components of the other electronic devices 200. For example, the memory 220 may include an image processing model 221.

[0060] Because the vehicles 10 travel in a designated formation, the cameras 240 of the other electronic devices 200 can capture images at specific times that the cameras 140 of the electronic device 100 cannot. According to one embodiment, the other electronic devices 200 can transmit information related to the images captured by the cameras 240 and / or information related to the external objects 30 identified from the images to the electronic device 100. The electronic device 100 can identify the surrounding environment of the convoy based on the information received from the other electronic devices 200 and control the travel of the convoy based on the surrounding environment.

[0061] According to one embodiment, the camera 140 may include a plurality of cameras 141, 142, and 143. For example, the camera 140 may include a first camera 141, a second camera 142, and / or a third camera 143, but is not limited thereto. According to one embodiment, the first camera 141, the second camera 142, and / or the third camera 143 may have different viewing angles from each other. For example, the first camera 141 may include a first image based on a first viewing angle (e.g., Figure 6A For example, the second camera 142 may include a lens for acquiring a second image based on a second perspective (for example: Figure 6B For example, the third camera 143 may include a lens for acquiring a third image based on a third perspective (for example: Figure 6C The first perspective, the second perspective, and the third perspective may be different from each other.

[0062] Figure 3A The diagram schematically illustrates the field of view according to the viewing angle of the camera. Figure 3B Shows images according to viewing angles.

[0063] Reference Figure 3A Camera 300 can capture images and / or images as part of an image set based on a viewing angle determined by the focal length of the lens. Focal length refers to the distance from the lens to the image formed on the film. The viewing angle can be determined based on the focal length. A shorter focal length results in a wider viewing angle, while a longer focal length results in a narrower viewing angle.

[0064] For example, when the focal length is approximately 7.5 mm, the viewing angle 301 may be approximately 180 degrees. When the focal length is approximately 28 mm, the viewing angle 302 may be approximately 75 degrees. When the focal length is approximately 50 mm, the viewing angle 303 may be approximately 47 degrees. When the focal length is approximately 105 mm, the viewing angle 304 may be approximately 23 degrees. When the focal length is approximately 135 mm, the viewing angle 305 may be approximately 18 degrees. When the focal length is approximately 300 mm, the viewing angle 306 may be approximately 8 degrees. When the focal length is approximately 500 mm, the viewing angle 307 may be approximately 5 degrees. When the focal length is approximately 1000 mm, the viewing angle 308 may be approximately 2 degrees.

[0065] The cameras 300 can be differentiated based on their focal lengths. For example, a camera with a focal length of approximately 7 mm to 15 mm may be a camera including a fisheye lens. A camera with a focal length of approximately 15 mm to 35 mm may be a camera including a wide-angle lens. A camera with a focal length of approximately 38 mm to approximately 55 mm may be a camera including a standard lens. A camera with a focal length of approximately 70 mm to approximately 1000 mm may be a camera including a telephoto lens.

[0066] Figure 3B The image 300a shows an example of an image acquired by a camera with a focal length of about 24 mm. Figure 3B The image 300b shows an example of an image acquired by a camera having a focal length of about 35 mm. Figure 3B Image 300c shows an example of an image acquired by a camera with a focal length of approximately 50 mm. Figure 3B Image 300d shows an example of an image acquired by a camera with a focal length of approximately 100 mm.

[0067] Reference Figure 3B The viewing angle can determine the range of the image captured by the camera 300. The wider the viewing angle, the wider the shooting range of the camera 300, and the smaller the photographed object. The shorter the viewing angle, the narrower the shooting range, and the larger the photographed object. For example, from Figure 3B The image 300a can obtain information about the largest number of subjects, but since the shooting range is very wide, it may be difficult to accurately distinguish the subjects. Figure 3B Image 300d allows for easy and accurate subject identification, but due to the narrow capture range, minimal subject information may be captured. The effects of radial distortion may vary depending on the viewing angle. For example, at wide viewing angles, radial distortion is more pronounced, potentially distorting the subject's precise position.

[0068] As reference Figure 3A and Figure 3B As mentioned above, the amount of information that can be obtained from the image and the accuracy of the information can vary depending on the characteristics of the camera 300. For example, depending on the electronic device (e.g. Figure 2 The camera of the electronic device 100 (for example: Figure 2 The camera 140 can identify external objects (e.g., Figure 1 The characteristics of the external object 30 may be different. In order to accurately identify the external object 30, an image based on a relatively small viewing angle may be required. In order to obtain the vehicle (for example: Figure 1 Various information about the surrounding environment of the vehicle 10) may require images based on a relatively large viewing angle.

[0069] According to one embodiment, the electronic device 100 captures or generates images based on different perspectives and identifies an external object 30 from the images, thereby accurately and diversely identifying the external object 30 and the surrounding environment. The electronic device 100 transmits at least a portion of the images to another electronic device 200 and identifies the external object 30 based on information related to the external object 30 captured by the electronic device 100 (e.g., first information) and information related to the external object 30 provided by the other electronic device 200 (e.g., second information).

[0070] Figure 4 is a flowchart illustrating the operation of an electronic device for recognizing an external object. Figure 5 Shows vehicles traveling in a convoy. Figure 6A 、 Figure 6B and Figure 6C Shows the front image and the cropped image.

[0071] Figure 4 The actions described in are performed by the processor (for example: Figure 1 The processor 110) executes the data in the memory (for example: Figure 2 When the instructions stored in the memory 120) are executed, the electronic device (for example: Figure 2 action of the electronic device 100).

[0072] Reference Figure 4In action 401, the electronic device 100 may use the camera 140 to obtain the leading vehicle (e.g. Figure 5 The front image in front of the leading vehicle 11).

[0073] For example, when executing the instructions stored in the memory 120, the processor 110 may obtain the front image of the front of the leading vehicle 11 captured by the camera 140. Figure 5 , the leading vehicle 11 may be located in front of the following vehicle 12. The camera 140 disposed in the leading vehicle 11 may be disposed in front of the leading vehicle 11 so as to capture the front of the leading vehicle 11. While the leading vehicle 11 is traveling, the camera 140 may generate a front image by capturing the front of the leading vehicle 11. The processor 110 may obtain the front image generated by the camera 140.

[0074] In action 402 , the electronic device 100 may generate a cropped image by cropping the front image.

[0075] For example, when executing the instructions stored in the memory 120, the processor 110 crops the front image acquired by the camera 140 into different sizes, thereby generating a plurality of cropped images. According to one embodiment, the cropped images may be generated based on the vanishing point of the front image (e.g., Figures 6A to 6C The cropped image is generated by using the vanishing point 600 of FIG.

[0076] Figure 6A The first image 601 shown is an example of a front image. The first image 601 may include external objects (eg, other vehicles 20, lane lines 31 and / or lanes 32). Figure 6A , the first image 601 may be an image based on a first perspective. For example, the first perspective may be wider than the second perspective and the third perspective described later. For example, a camera (e.g.: Figure 2 The camera 140 may be an ultra-wide-angle camera for a wide shooting range, but is not limited thereto. The vanishing point 600 of the front image is the point where parallel straight lines in the actual physical space converge when they point in the same direction. The processor (e.g., Figure 2 Processor 110 can identify vanishing point 600 in the front image and set a region of interest (ROI) based on vanishing point 600. A ROI can be an area within a certain range relative to vanishing point 600 in the image. For example, processor 110 can set a first region 600a and a second region 600b relative to vanishing point 600. For example, first region 600a can be larger than second region 600b.

[0077] Figure 6B The image shown as a cropped image can be a first image based on a first perspective (for example: Figure 6A The second image 602 is a first image that crops the first area 600a from the first image 601. The second image 602 is a first image that crops the first area (e.g., Figure 6A The cropped image 602 is generated based on the first area 600a of the image 601, and therefore has a narrower range than the first image 601. For example, the second image 602 can be an image based on a second perspective that is narrower than the first perspective. For example, when the camera 140 is an ultra-wide-angle camera, the second image 602 can be an image corresponding to a standard perspective, but is not limited thereto. The second image 602 is based on a second perspective that is narrower than the first perspective, so it displays an area narrower than the first image 601, but can display objects included in the first image larger. For example, a portion of external objects included in the first image 601 (for example, other vehicles 20, lane markings 31 and / or lanes 32) can be enlarged and displayed in the second image 602.

[0078] Figure 6C The image shown as a cropped image can be a first image based on a first perspective (for example: Figure 6A The first image 601) is cropped to the second area (for example: Figure 6A The third image 603 is a cropped image generated by cropping the second area 600b from the first image 601, and thus may have a larger image quality than the first image 601 and the second image (e.g., Figure 6B For example, the third image 603 may be an image based on a third perspective that is narrower than the first perspective and the second perspective. For example, when the camera (e.g. Figure 2 When the camera 140 (e.g., the camera 140) is an ultra-wide-angle camera, the third image 603 may be an image corresponding to a narrower angle of view than the standard angle of view, but is not limited thereto. Third image 603 is based on a third angle of view that is narrower than the first and second angles of view, and therefore displays a narrower area than first image 601 and second image 602. However, it may display a portion of external objects (e.g., other vehicles 20, lane markings 31, and / or lanes 32) included in first image 601 and second image 602 more enlarged. For example, external object 30 included in first image 601 and second image 602 may be displayed more magnified in third image 603. Figure 6B and Figure 6C The second image 602 and the third image 603 shown are only exemplary, and the cropped image is not limited thereto. Alternatively, the first image 601, the second image 602, and the third image 603 may also be captured by different cameras (e.g., Figure 2 images captured by the first camera 141, the second camera 142 and the third camera 143).

[0079] In action 403 , the electronic device 100 may transmit at least one of the front image and the cropped image to each other electronic device 200 configured in each following vehicle 12 .

[0080] For example, refer to Figure 5 When executing instructions stored in memory 120, processor 110 may utilize wireless communication devices (e.g., Figure 2 The wireless communication device 130 of the embodiment of the present invention transmits data including the front image and / or the cropped image to each other electronic device 200. The processor 110 may transmit both the front image and the cropped image to each other electronic device 200, or may transmit a portion of the front image or the cropped image to each other electronic device 200.

[0081] According to one embodiment, in order to reduce the capacity of data, the processor 110 may change the size of the front image or cropped image to be sent and send it. As the resized front image or cropped image is sent, the amount of processing by the wireless communication device 130 can be reduced. According to one embodiment, the processor 110 may change the size of the front image or cropped image based on the size of the neural network model input to the other electronic device 200. Since high-resolution (e.g., 4K, Full High Definition (FHD)) images are not required when the other electronic device 200 recognizes the external object 30 from the front image or cropped image, the processor 110 may compress the front image or cropped image into an image below HD and send it.

[0082] In action 404 , the electronic device 100 may obtain first information from the external object 30 included in at least the other of the front image and the cropped image.

[0083] For example, when executing instructions stored in the memory 120 , the processor 110 may identify the external object 30 included in at least one of the front image and the cropped image, and obtain first information related to the external object 30 .

[0084] For example, the electronic device 100 can recognize the external object 30 from the first image 601 as the front image. Figure 6A, the electronic device 100 can recognize the external object 30 included in the first image 601. The first image 601 is an image based on the widest viewing angle and therefore can include the widest range of surrounding environment. Since the first image 601 captured by the camera with the widest viewing angle includes the most information, the electronic device 100 may include an AI accelerator such as a neural network processing unit (NPU) and / or a graphic processing unit (GPU) with higher performance than other electronic devices 200 for computer vision processing based on artificial intelligence (AI).

[0085] According to one embodiment, the electronic device 100 may recognize the external object 30 using a neural network model that has been pre-trained to recognize the external object 30 from an image corresponding to the external object 30, and obtain first information related to the external object 30. For example, the first information may include information related to the type (class), location, size, and / or current state of the external object 30.

[0086] In action 405 , the electronic device 100 may obtain second information from the other electronic device 200 .

[0087] For example, when executing instructions stored in the memory 120 , the processor 110 may receive, from the other electronic device 200 , second information related to the external object 30 obtained from the external object 30 included in at least one of the front image and the cropped image.

[0088] For example, the other electronic device 200 may recognize the external object 30 from the second image 602 and / or the third image 603 which are cropped images. Figure 6B and Figure 6C , the other electronic device 200 can recognize the external object 30 included in the second image 602 and / or the third image 603. The second image 602 and / or the third image 603 are images based on a relatively narrow viewing angle, and therefore can magnify the external object 30. Since the external object 30 is displayed more accurately in the second image 602 and / or the third image 603, the other electronic device 200 can use the neural network model to more accurately recognize the external object 30.

[0089] For example, when it is difficult to identify the external object 30 in the first image 601 because the external object 30 is displayed too small, the external object 30 can be identified in the second image 602 and / or the third image 603. According to one embodiment, the other electronic device 200 can use a neural network model pre-trained in a manner of identifying the external object 30 from an image corresponding to the external object 30 to identify the external object 30 and obtain second information related to the external object 30. For example, the second information may include information related to the type, position, size and / or current state of the external object 30. Figure 5 , the other electronic device 200 can send the second information to the electronic device 100, and the electronic device 100 can receive the second information from the other electronic device 200.

[0090] In action 406 , the electronic device 100 may recognize the external object 30 based on the first information and the second information.

[0091] For example, when executing instructions stored in the memory 120, the processor 110 can identify the external object 30 located in front of the lead vehicle 11 by integrating the first information and the second information. Since the first information includes a lot of information and the second information includes accurate information, the electronic device 100 can accurately identify the surrounding environment based on the first information and the second information.

[0092] When the electronic device 100 analyzes all the front images and the cropped images in order to recognize the external object 30, it may take a relatively long time due to the large amount of data to be processed. The processing of data by the electronic device 100 may require high performance. According to one embodiment, the electronic device 100 analyzes at least one of the front image and the cropped image (e.g., Figure 6A The electronic device 100 recognizes the external object 30 based on the first image 601 (of the front image), while the other electronic device 200 recognizes the external object 30 based on at least one of the front image and the cropped image (e.g., the second image 602 and / or the third image 603). This reduces the data processing burden on the electronic device 100. According to one embodiment, the electronic device 100 recognizes the external object 30 based on the first information and the second information, thereby accurately recognizing a wide range of information about the surrounding environment. In addition, the data transmitted by the electronic device 100 to the other electronic device 200 can be a cropped image cropped from the front image, rather than the front image itself.

[0093] Hereinafter, the operation of the electronic device 100 will be described in more detail.

[0094] Figure 7A is a flowchart illustrating an operation of an electronic device for recognizing an external object by transmitting a cropped image to another electronic device.

[0095] Figure 7AThe actions described in may be performed by a processor (e.g.: Figure 2 The processor 110 and the processor 210) execute in the memory (for example: Figure 2 When the instructions stored in the memory 120 and the memory 220) cause the electronic device (for example: Figure 2 Hereinafter, the other electronic device 200 will be described as the first following vehicle (for example, Figure 9 The first other electronic device 201 configured in the first following vehicle 911 of the following vehicle 12 and the second following vehicle (for example: Figure 9 The second other electronic device 202 is configured in the second following vehicle 912, but is not limited thereto.

[0096] Reference Figure 7A In action 701, the electronic device 100 may use the camera 140 to obtain a first image (eg, Figure 6A 601).

[0097] For example, the first image 601 may be a front image of the front of the leading vehicle 11. Action 701 may correspond to Figure 4 For example, when executing the instructions stored in the memory 120 , the processor 110 may obtain a front image of the front of the leading vehicle 11 captured by the camera 140 .

[0098] In action 702, the electronic device 100 may generate a second image (eg, Figure 6B The second image 602) and the third image (for example: Figure 6C 603).

[0099] For example, the second image 602 and the third image 603 may be cropped images generated by cropping some areas of the first image 601. As described above, the second image 602 may be a first area (e.g., Figure 6A The third image 603 may be a cropped image generated by cropping the first area 600a of the first image 601 (for example: Figure 6A The cropped image is generated by cropping the second area 600b). Action 702 may correspond to Figure 4 For example, when executing the instructions stored in the memory 120 , the processor 110 may crop the first image 601 acquired by the camera 140 into different sizes, thereby generating the second image 602 and the third image 603 .

[0100] In action 703 , the electronic device 100 may send the cropped images (eg, the second image 602 and the third image 603 ) to each of the other electronic devices 200 .

[0101] For example, when executing the instructions stored in the memory 120, the processor 110 may send the second image 602 to the first other electronic device 201 and the third image 603 to the second other electronic device 202. Action 703 may correspond to Figure 4 Action 403.

[0102] In action 704, the electronic device 100 may select an external object (eg, Figure 1 The third information is obtained from the external object 30).

[0103] For example, the third information may correspond to the aforementioned first information. For example, when executing the instructions stored in the memory 120, the processor 110 may obtain the third information related to the external object 30 from the external object 30 included in the first image 601 as the front image. Action 704 may correspond to Figure 4 Action 404.

[0104] In action 705 , the first other electronic device 201 may obtain fourth information from the external object 30 included in the second image 602 .

[0105] In action 706 , the second other electronic device 202 may obtain fifth information from the external object 30 included in the third image 603 .

[0106] For example, the processor 210 of the first other electronic device 201 can obtain fourth information related to the external object 30 from the second image 602. The processor 210 of the second other electronic device 202 can obtain fifth information related to the external object 30 from the third image 603. The second image 602 and / or the third image 603 are images based on a relatively narrow viewing angle, and therefore can magnify the external object 30. Because the external object 30 is displayed more accurately in the second image 602 and / or the third image 603, the first other electronic device 201 and the second other electronic device 202 can use the neural network model to more accurately recognize the external object 30. The aforementioned second information may include the fourth information and the fifth information.

[0107] In action 707 , the electronic device 100 may receive fourth information and fifth information from the other electronic device 200 .

[0108] For example, when executing the instructions stored in the memory 120, the processor 110 may receive fourth information from the first other electronic device 201 and fifth information from the second other electronic device 202. Action 707 may correspond to Figure 4 Action 405.

[0109] In action 708 , the electronic device 100 may recognize the external object 30 based on the third information, the fourth information, and the fifth information.

[0110] For example, when executing instructions stored in memory 120, processor 110 may identify external object 30 based on third information acquired from first image 601, fourth information received from first other electronic device 201, and fifth information received from second other electronic device 202. The third information may include various forward information, but when external object 30 appears too small in first image 601, accurate identification of external object 30 may be difficult. The fourth information based on second image 602 may include less information than the third information, but may include more accurate information about external object 30 than the third information. The fifth information based on third image 603 may include less information than the third and fourth information, but may include more accurate information about external object 30 than the third and fourth information. Processor 110 identifies external object 30 located in front of lead vehicle 11 based on the third, fourth, and fifth information, thereby accurately identifying various external objects 30.

[0111] In action 709 , the electronic device 100 may control the vehicle fleet to travel based on the recognized external object 30 .

[0112] For example, when executing instructions stored in memory 120, processor 110 may utilize a neural network model to control the movement of the convoy. For example, when the recognized external object 30 is a traffic light, processor 110 may control the movement of the convoy based on the signal from the traffic light. For example, when the recognized external object 30 is an obstacle, processor 110 may determine a driving route to avoid the obstacle.

[0113] According to one embodiment, the electronic device 100 transmits a cropped image of a front image to another electronic device 200, and recognizes an external object 30 from the front image and the cropped image. The front image can be analyzed by the electronic device 100, and the cropped image can be analyzed by the other electronic device 200. This reduces the amount of data processing by the electronic device 100 and allows accurate recognition of many external objects 30.

[0114] In the aforementioned example, the electronic device 100 is described as generating a cropped image from the front image and transmitting the cropped image to the other electronic device 200, but the present invention is not limited thereto. Alternatively, the electronic device 100 may transmit all the front images and the cropped images to the other electronic device 200. The electronic device 100 and the other electronic device 200 may process either the front image or the cropped image as the main image and the remaining images as sub-images to recognize the external object 30.

[0115] Figure 7B This is a flowchart illustrating the operation of an electronic device that recognizes external objects through distributed computing or edge computing.

[0116] According to one embodiment, in order to identify external objects (e.g. Figure 6A Distributed computing and / or edge computing can be used to identify external objects 30. For example, electronic device 100 and other electronic devices 201 and 202 can use distributed computing and / or edge computing to process first image 601, a forward image captured by a camera of lead vehicle 11, and second image 602 and third image 603 generated from the first image. The following describes the operation of electronic device 100 in identifying external objects using distributed computing or edge computing. Figure 7B The actions recorded in can be memory (for example: Figure 2 The instructions stored in the memory 120) are executed by the processor (e.g.: Figure 1 When executed by the processor 110 of the electronic device (for example: Figure 2 The operation of the electronic device 100). Figure 7B In action 711 , the electronic device 100 may identify the first other electronic device 201 and the second other electronic device 202 connected to the network.

[0117] The network may refer to a communication network for exchanging information between the electronic device 100 and other electronic devices 201 and 202. The electronic device 100 may send a cropped image to other electronic devices 201 and 202 connected to the network. The other electronic devices 201 and 202 may send information (e.g., the fourth information or the fifth information) to the electronic device 100 connected to the network. The electronic device 100 may recognize the presence of other electronic devices 201 and 202 connected to the network.

[0118] In action 712 , the electronic device 100 may identify capability information of the identified other electronic devices 201 , 202 .

[0119] The electronic device 100 and the other electronic devices 201 and 202 process the first image 601 , which is a front image obtained by a camera of the leading vehicle 11 , and the second image 602 and the third image 603 generated from the first image, through distributed computing and / or edge computing.

[0120] For example, the electronic device 100 of the leading vehicle 11 can obtain information that can identify the capabilities of each electronic device by communicating with the first other electronic device 201 and the second other electronic device 202 of the following vehicle 12 connected to the wireless network. The capability information that can identify the capabilities of each electronic device may include the number of cores of the processor installed in the electronic device, the CPU clock speed, memory bandwidth, CPU architecture, the number of stream processors of the GPU, GPU architecture, GPU clock speed, memory bandwidth, floating point operations per second (FLOPS), and operations per second (OPS) as performance information of the NPU.

[0121] In action 713 , the electronic device 100 may allocate work to each of the other electronic devices 201 , 202 according to the identified capability information.

[0122] In one embodiment of the present invention, the electronic device 100 of the lead vehicle 11 utilizes the capability information identified for the first and second other electronic devices 201, 202 to enable the first and second other electronic devices 201, 202 to perform distributed AI-based computer vision processing of the second and third images generated from the first image captured by the camera, respectively. For example, the electronic device 100 may allocate the AI-based computer vision processing of the second image to the first other electronic device 201, and allocate the AI-based computer vision processing of the third image to the second other electronic device 202.

[0123] In action 714 , the electronic device 100 may obtain a processing result processed by each of the other electronic devices 201 , 202 .

[0124] According to one embodiment, the electronic device 100 may obtain an AI-based computer vision processing result of the second image from the first other electronic device 201, and obtain an AI-based computer vision processing result of the third image from the second other electronic device 202. For example, the first other electronic device 201 may obtain fourth information from an external object included in the second image and provide the fourth information to the electronic device 100. For example, the second other electronic device 202 may obtain fifth information from an external object included in the third image and provide the fifth information to the electronic device 100.

[0125] In action 715 , the electronic device 100 may recognize the external object using the processing results acquired from each of the other electronic devices 201 , 202 .

[0126] According to one embodiment, the electronic device 100 may analyze the visual processing results of the second image acquired from the first other electronic device 201 and the visual processing results of the third image acquired from the second other electronic device 202. The electronic device 100 may be configured to recognize external objects by combining the visual processing results of the first image processed by the electronic device 100 and the visual processing results of the second image and the third image processed by the other electronic devices 201 and 202. The electronic device 100 may provide the processing results to the user.

[0127] In one embodiment of the present invention, the electronic device 100 can transmit the second image to the first other electronic device 201 so that the first other electronic device 201 obtains fourth information from the external object included in the second image, and transmit the third image to the second other electronic device 202 so that the second other electronic device 202 obtains fifth information from the external object included in the third image.

[0128] The action of this electronic device 100 can be equivalent to a distributed computing method in which the workload of AI-based computer vision processing, such as detection and recognition of objects in images between the electronic device 100 connected to the network, the first other electronic device 201 and the second other electronic device 202, is executed on each electronic device.

[0129] In other words, the electronic device 100 of the leading vehicle 11 can be controlled so that the AI-based computer vision processing work such as object detection in the first image 601 as the front image obtained by the camera 140 of the leading vehicle 11 is distributed among the first other electronic device 201 and the second other electronic device 202, thereby also reducing the workload of the electronic device 100.

[0130] Furthermore, in one embodiment of the present invention, distributed computing and / or edge computing can be used to process the first image 601 of the front image obtained by the camera of the leading vehicle 11, the second image 602 and the third image 603 generated from the first image of the electronic device 100, the first other electronic device 201 and the second other electronic device 202.

[0131] In order to perform image analysis through this distributed computing / edge computing method, the electronic device 100 can distribute work among the electronic device 100, the first other electronic device 201 and the second other electronic device 202, and can perform control to adjust the allocation of computing resources in order to complete the specified work.

[0132] In this way, according to an embodiment of the present invention, the electronic device 100 can implement various technologies that make it easy to share computing workloads among multiple interconnected electronic devices (the first other electronic device 201 and the second other electronic device 202) through distributed computing.

[0133] In particular, in AI-based computer vision processing tasks that require high computing power, the computational requirements required to train and deploy machine learning models are high. Therefore, through distributed computing capabilities, that is, distributing computing work across multiple electronic devices (nodes), large-scale data sets and complex algorithms can be processed more efficiently. Therefore, when distributed computing is performed between electronic devices 100, 201, and 202 connected to the network, as in an embodiment of the present invention, the training process can be accelerated and AI applications can achieve real-time decision-making.

[0134] Figure 7C The present invention is a signal flow diagram for explaining the steps of an electronic device of a vehicle discovering other electronic devices of other vehicles for requesting work sharing according to a distributed computing method.

[0135] Figure 7CThe electronic device 720 serves as a central entity that performs the following roles: it distributes the work for identifying information required for vehicle driving (e.g., external objects, etc.) (e.g., computing operations for identifying objects from image data acquired through a camera, etc.) to other electronic devices 722, 724, 726 in a distributed computing manner, and can perform the role of a master device of a distributed computing system structure. Therefore, in order to distribute work in a distributed computing manner, the electronic device 720 according to one embodiment performs the role of controlling, coordinating or managing the other electronic devices 722, 724, 726 (i.e., slave devices). Specifically, in order to distribute work in a distributed computing manner, the electronic device 720 distributes work to the slave devices (other electronic devices 722, 724, 726) and collects the output of the computing operation results performed by the slave devices, thereby effectively distributing the workload.

[0136] When electronic device 720 generates data to be processed by distributed computing in action 730, it is configured to broadcast discovery messages 732a, 732b, and 732c in order to discover electronic devices capable of performing distributed computing operations in action 732. In action 732, electronic device 720 broadcasts discovery messages 732a, 732b, and 732c, thereby initiating a process of scanning for other electronic devices 722, 724, and 726 located around electronic device 720.

[0137] At this time, in order to select a discovery resource, the electronic device 720 according to an embodiment may collectively sense the entire discovery resource pool.

[0138] exist Figure 7C In response, the first other electronic device 722 that receives the discovery message 732a sends a discovery confirmation (Discovery Ack. (Acknowledgment)) message 734a to the electronic device 720. The discovery confirmation message 734a may include at least one of unique identification information (Identification information) assigned to enable the electronic device 720 to identify the first other electronic device 722 in the network, location information of the first other electronic device 722, and a key value (Key value) to be used for encryption (Encryption) / decryption (Decryption) of data sent and received between the first other electronic device 722 and the electronic device 720.

[0139] In response, the second other electronic device 724 that receives the discovery message 732b may send a discovery confirmation message 734b to the electronic device 720. The discovery confirmation message 734b may include at least one of unique identification information assigned to enable the electronic device 720 to identify the second other electronic device 724 in the network, location information of the second other electronic device 724, and a key value to be used for encryption / decryption of data sent and received between the second other electronic device 724 and the electronic device 720.

[0140] In response, the third other electronic device 726 that has received the discovery message 732c may send a discovery confirmation message 734c to the electronic device 720. The discovery confirmation message 734c may include at least one of unique identification information assigned to enable the electronic device 720 to identify the third other electronic device 726 in the network, location information of the third other electronic device 726, and a key value to be used for encryption / decryption of data sent and received between the third other electronic device 726 and the electronic device 720.

[0141] The electronic device 720 that receives the discovery confirmation messages 734a, 734b, and 734c may use the information included in the discovery confirmation messages 734a, 734b, and 734c to identify the other electronic devices 722, 724, and 726 in action 736. Furthermore, according to one embodiment, in order to allocate the distributed computing work of the data generated in action 730 to the identified other electronic devices 722, 724, and 726, the electronic device 720 preferably confirms information such as processor information, computing power, storage space size, and memory performance (memory bandwidth, memory size) of the identified other electronic devices 722, 724, and 726.

[0142] According to an embodiment, the processor information may include architecture information of the processor, CPU information, GPU information, NPU information, and cache information of the processor.

[0143] Thus, the electronic device 720 can send capability request messages (Capability Request messages) 738a, 738b, 738c identified in the action 736 for confirming the capability information of the other electronic devices 722, 724, 726 to the first other electronic device 722, the second other electronic device 724 and the third other electronic device 726 respectively.

[0144] In response, the first other electronic device 722, the second other electronic device 724 and the third other electronic device 726 that receive the capability request messages 738a, 738b, 738c may send capability response messages (Capability Response message) 740a, 740b, 740c including their own capability information to the electronic device 720, respectively.

[0145] The capability request message may include the information shown in Table 1.

[0146] [Table 1]

[0147]

[0148]

[0149] The electronic device 720 that receives the capability response messages 740a, 740b, and 740c may, in action 742, identify the capability information of the other electronic devices 722, 724, and 726. Furthermore, in action 744, the electronic device 720 may identify the electronic devices to be shared with based on the identified capability information. For example, the electronic device 720 may use the capability information of the other electronic devices 722, 724, and 726 included in the capability response messages to determine whether the other electronic devices 722, 724, and 726 are capable of performing distributed computing tasks, and the degree of shared distributed computing tasks (workload).

[0150] To this end, in one embodiment, the other electronic devices 722, 724, and 726, as shown in Table 1, include the information element (IE) required by the electronic device 720 in the capability request message as their respective capability information in the capability response messages 740a, 740b, and 740c.

[0151] In the action 744, the electronic device 720 that identifies the electronic device to be shared based on the identified capability information transfers the work assigned to the first other electronic device 722 (action 746a) and transfers the work assigned to the second other electronic device 724 (action 746b). Figure 7C As shown in FIG, after identifying the electronic devices to be shared with the work in action 744 , the electronic device 720 transfers the allocated work to only the first other electronic device 722 and the second other electronic device 724 except the third other electronic device 726 . Figure 7CAn embodiment is shown in which the electronic device 720 identifies in action 744 that the result of information included in the capability response message received from the third other electronic device 726 satisfies any one of the following conditions, the conditions including: condition 1, the capability of the third other electronic device 726 cannot meet the minimum requirements for being able to perform work sharing through distributed computing; condition 2, the third other electronic device 726 cannot perform work sharing because it is currently performing other work; and condition 3, work sharing through distributed computing can be successfully completed only through the first other electronic device 722 and the second other electronic device 724.

[0152] Upon receiving the assigned work from electronic device 720, first other electronic device 722 and second other electronic device 724 utilize their respective computing capabilities to execute distributed computing work 748a and 748b, respectively, to complete the assigned task. Upon completion of the distributed computing work, as shown in actions 750a and 750b, first other electronic device 722 and second other electronic device 724 each transmit the work processing results to electronic device 720.

[0153] The electronic device 720 that receives the work processing results from the first other electronic device 722 and the second other electronic device 724 performs data processing in action 752 using the work processing result (distributed computing work 1) which is the result of the computing operation related to the work assigned to itself, the work processing result (distributed computing work 2) which is the result of the computing operation transmitted from the first other electronic device 722, and the work processing result (distributed computing work 3) which is the result of the computing operation transmitted from the second other electronic device 724.

[0154] Figure 7C The action flow (action 730 to action 752) of the electronic device 720 shown from the time point of generating the data to be processed by distributed computing to the time point of completing the data processing by using the work results of the distributed computing processing performed by each electronic device can be performed by Figure 7C The electronic device 720 of the main role of the distributed computing system shown is controlled to execute within a certain period T (770).

[0155] And, in Figure 7C In the process of sending a discovery message and receiving a discovery confirmation message by the electronic device 720 in order to discover other electronic devices to perform distributed computing tasks can be called a scanning process. Figure 7CThe figure only shows that other electronic devices 722, 724, and 726 send discovery confirmation messages 734a, 734b, and 734c to electronic device 720 in response to discovery messages 732a, 732b, and 732c. However, this is only one embodiment. Other electronic devices 722, 724, and 726 may also send a discovery negative confirmation message (Discovery Not-Acknowledgment Message, Discovery Nack.Msg) to reject the distributed computing work request. In this case, the other electronic devices that receive the discovery message may determine the discovery negative confirmation message based on their own computing capabilities, current workload, network environment, resource information, etc.

[0156] According to the above embodiment, in order to perform distributed computing between the electronic devices 720, 722, 724, 726, wireless resource allocation for direct communication between the electronic devices 720, 722, 724, 726 may also be required. Figure 19A 、 Figure 19B 、 Figure 19C and Figure 19D Wireless resource allocation for direct communication between electronic devices 720 , 722 , 724 , and 726 according to one embodiment is described.

[0157] The electronic device 100 for performing the above-mentioned operations will be described below.

[0158] Figure 8 FIG. 1 is a flowchart illustrating an operation of an electronic device recognizing an external object based on a main image and a sub-image according to an embodiment. Figure 9 Schematically showing the Figure 8 The actions shown in FIG. 1 are the process of processing the main image and the sub-image.

[0159] Figure 8 The actions recorded in can be memory (for example: Figure 2 The instructions stored in the memory 120) are executed by the processor (e.g.: Figure 2 When executed by the processor 110 of the electronic device (for example: Figure 2 action of the electronic device 100).

[0160] According to one embodiment, the front image and the cropped image can be distinguished as a main image and a sub-image. In the present disclosure, the main image may refer to an image processed by the electronic device 100 or other electronic device 200 based on a relatively high frame rate (Frame Per Second, FPS). In the present disclosure, the sub-image is a residual image different from the main image, and may refer to an image processed by the electronic device 100 or other electronic device 200 based on a relatively low FPS. For example, when all images are processed by the processor 110 or the processor 210 at the same FPS, external objects (e.g., Figure 1 The recognition speed of an external object 30 may be reduced. Therefore, to alleviate this problem, the image can be divided into a main image and a sub-image. The electronic device 100 can recognize the external object 30 by differently allocating the main image to be processed by the processor 110 and the processor 210 and combining the first information and the second information.

[0161] Reference Figure 8 In action 801 , the electronic device 100 may determine a first main image to be processed as a main image and a first sub-image to be processed as a sub-image.

[0162] For example, when executing instructions stored in the memory 120, the processor 110 may distinguish the front image and the cropped image into a first main image processed by the processor 110 based on a first FPS and a first sub-image processed by the processor 110 based on a second FPS. The first main image may refer to an image processed as a main image by the electronic device 100. The first sub-image is at least a portion of the remaining images other than the main image, and may refer to an image processed as a sub-image by the electronic device 100. For example, the first FPS may be 20 FPS to 40 FPS, and the second FPS may be 3 FPS to 5 FPS, but is not limited thereto. In the present disclosure, the action of determining the main image and the sub-image may refer to the action of assigning each image to a main image or a sub-image.

[0163] For example, the processor 110 may determine the front image as the first main image and the cropped image as the second main image. For example, the processor 110 may determine the first image 601 as the first main image and the second image 602 and the third image 603 as the second sub-images. The first image 601 may be processed by the processor 110 based on a first FPS, and the second image 602 and the third image 603 may be processed by the processor 110 based on a second FPS. Since the first image 601 is processed based on the first FPS, it can serve as the primary means for identifying the external object 30 included in the first image 601. Since the second image 602 and the third image 603 are processed based on the second FPS, they can serve as auxiliary means for identifying the external object 30 included in the second image 602 and the third image 603.

[0164] In action 802 , the electronic device 100 may determine a second main image processed as a main image and a second sub-image processed as a sub-image by another electronic device.

[0165] For example, when executing instructions stored in the memory 120, the processor 110 may distinguish the front image and the cropped image into a second main image processed by the processor 210 of each other electronic device 200 based on a first FPS and a second sub-image processed by the processor 210 based on a second FPS. The second main image may refer to an image processed as a main image by the other electronic device 200. The second sub-image is at least a portion of the remaining image other than the main image and may refer to an image processed as a sub-image by the other electronic device 200.

[0166] For example, the processor 110 may determine a cropped image as the second main image and the front image and the remaining images in the cropped image as the second sub-images. For example, the processor 110 may determine the second image 602 as the main image and the first image 601 and the third image 603 as sub-images. The second image 602 may be processed by the processor 210 based on the first FPS, and the first image 601 and the third image 603 may be processed by the processor 210 based on the second FPS. Since the second image 602 is processed based on the first FPS, it can serve as the primary means for identifying the external object 30 included in the second image 602. Since the first image 601 and the third image 603 are processed based on the second FPS, they can serve as auxiliary means for identifying the external object 30 included in the first image 601 and the third image 603. The second main image may be an image different from the first main image.

[0167] In action 803 , the electronic device 100 may obtain first information from the first main image and the first sub-image.

[0168] For example, when executing instructions stored in memory 120, processor 110 processes the first main image based on a first FPS and the second main image based on a second FPS, thereby obtaining first information related to external object 30. For example, when first image 601 is determined to be the first main image, processor 110 processes first image 601 based on the first FPS and processes second image 602 and third image 603 based on the second FPS, thereby obtaining first information. First image 601 is processed based on the first FPS and thus can have relatively high accuracy. Second image 602 and third image 603 are processed based on the second FPS and thus can have relatively low accuracy.

[0169] In action 804 , the electronic device 100 may receive second information from the other electronic device 200 .

[0170] For example, the other electronic device 200 processes the second main image based on the first FPS and the second sub-image based on the second FPS, thereby obtaining the second information. For example, when the second image 602 is determined to be the second main image, the processor 210 processes the second image 602 based on the first FPS and processes the first image 601 and the third image 603 based on the second FPS, thereby obtaining the second information. The second image 602 is processed based on the first FPS, so it can have a relatively high accuracy. The first image 601 and the third image 603 are processed based on the second FPS, so they can have a relatively low accuracy. When executing the instructions stored in the memory 120, the processor 110 can receive the second information obtained by the other electronic device 200.

[0171] In action 805 , the electronic device 100 may recognize the external object 30 based on the first information and the second information.

[0172] For example, when executing instructions stored in the memory 120 , the processor 110 may identify the external object 30 located in front of the lead vehicle 11 by integrating the first information and the second information.

[0173] According to one embodiment, the processor 110 may identify the external object 30 based on the main image and the sub-image using the first information and the second information. For example, the processor 110 may use the first information as a primary means and the second information as a supplementary means to identify the external object 30 included in the first image 601. The second information may be used to cross-validate the identification of the external object 30 included in the first image 601. For example, the processor 110 may use the second information as a primary means and the first information as a supplementary means to identify the external object 30 included in the second image 602. The first information may be used to cross-validate the identification of the external object 30 included in the second image 602.

[0174] According to one embodiment, when the first information and the second information are inconsistent, the processor 110 may identify the external object 30 based on the information with higher reliability. For example, when the external object 30 identified based on the first information is different from the external object 30 identified based on the second information, the processor 110 may identify the external object 30 based on the information with higher reliability.

[0175] For example, information obtained based on an image processed as a main image can have higher reliability than information obtained based on an image processed as a sub-image. When the external object 30 identified based on the first information is different from the external object 30 identified based on the second information, the processor 110 can identify the image including the external object 30. For example, the processor 110 can identify in which of the first image 601, the second image 602, and the third image 603 the external object 30 is included. The external object 30 can also be included in more than one image. The processor 110 can identify whether the image including the external object 30 is the first main image or the second image 602. When the image is identified as the first main image, the processor 110 can identify the external object 30 based on the first information. When the image is identified as the second main image, the processor 110 can identify the external object 30 based on the second information. According to one embodiment, the electronic device 100 identifies the external object 30 based on information with high reliability, thereby improving the accuracy of identifying the external object 30.

[0176] According to one embodiment, when the first information and the second information are inconsistent, the processor 110 may conservatively identify the external object 30. For example, if the distance between the external object 30 and the lead vehicle 11 is identified as 10 meters by the first information and as 5 meters by the second information, the processor 110 may identify the distance between the external object 30 and the lead vehicle 11 as 5 meters. Because the identification of the external object 30 is used to determine the driving route and driving control of the convoy, the processor 110 may conservatively identify the external object 30 to prevent safety accidents. As used herein, the processor 110 "conservatively identifies the external object 30" includes the action of determining, when the distance between the vehicle and the external object 30 identified by the first information (the first identification distance) and the distance between the vehicle and the external object 30 identified by the second information (the second identification distance) differ, that the processor 110 determines the distance between the first and second distances that has a lower probability of a collision or a vehicle accident as information used for driving control and autonomous driving of the vehicle. In other words, "conservative" may mean that when multiple different pieces of information are input, processor 110 selects the information that indicates a lower likelihood of an accident. For example, if the distance between external object 30 and leading vehicle 11 is determined to be 10 meters or 5 meters, electronic device 100 may be configured to identify the distance as 10 meters, thereby reducing the likelihood of an accident.

[0177] Figure 9 9 shows a case where the following vehicles include a first following vehicle 911 and a second following vehicle 912. Figure 9The electronic device 100 configured in the leading vehicle 11 can obtain the first image 601 and generate the second image 602 and the third image 603 from the first image 601. The electronic device 100 can send data including the first image 601, the second image 602, and the third image 603 to the first other electronic device 201 configured in the first following vehicle 911 and the second other electronic device 202 configured in the second following vehicle.

[0178] For example, the electronic device 100 processes the first image 601 as a main image (e.g., the first main image) and processes the second image 602 and the third image 603 as sub-images (e.g., the first sub-image), thereby obtaining information related to the external object 30. For example, a first other electronic device 201 disposed in the first following vehicle 911 processes the second image 602 as a main image (e.g., the second main image) and processes the first image 601 and the third image 603 as sub-images (e.g., the second sub-image), thereby obtaining information related to the external object 30. The first other electronic device 201 may transmit data including the information to the electronic device 100. For example, a second other electronic device 202 disposed in the second following vehicle may process the third image 603 as a main image (e.g., the third main image) and processes the first image 601 and the second image 602 as sub-images (e.g., the third sub-image), thereby obtaining information related to the external object 30. The second other electronic device 202 may transmit data including the information to the electronic device 100.

[0179] According to one embodiment, the electronic device 100 can recognize the external object 30 by integrating information. The image processed as the main image can be accurately analyzed by one of the electronic device 100, the first other electronic device 201, and the second other electronic device 202. The image processed as a sub-image by a portion of the electronic device 100, the first other electronic device 201, and the second other electronic device 202 can be used to cross-verify information obtained by processing the image as the main image.

[0180] Figure 10 This is a flowchart illustrating the operation of an electronic device using multiple cameras.

[0181] In the above example, the electronic device 100 is described as performing an action of generating a cropped image from a front image and sending the front image and / or the cropped image to the other electronic device 200, but is not limited thereto. Alternatively, the camera 140 may include multiple cameras 141, 142, 143 for shooting images based on different perspectives. For example, the electronic device 100 may send at least a portion of the front image acquired by each of the multiple cameras 141, 142, 143 to the other electronic device 200. The other electronic device 200 may use the front image to acquire information related to an external object (e.g., Figure 1 The electronic device 100 may obtain second information related to an external object 30 (e.g., an external object 30) and transmit the obtained second information to the electronic device 100. The electronic device 100 may recognize the external object 30 based on the first information obtained through at least one of the front images and the second information received from the other electronic device 200. The following describes an electronic device 100 including multiple cameras 141, 142, and 143. The electronic device 100 described later may include substantially the same components as the aforementioned electronic device 100, and repeated descriptions may be omitted.

[0182] Figure 10 The actions described in the can be when the memory (for example: Figure 2 The instructions stored in the memory 120 and the memory 220) are executed by the processor (for example: Figure 2 When the processor 110 and the processor 210 of the electronic device (for example: Figure 2 The actions of the electronic device 100 and other electronic devices 200).

[0183] Reference Figure 10 In action 1001, the electronic device 100 may acquire a first image (eg, Figure 9 The first image 601), the second image (for example: Figure 9 ), and a third image (e.g., Figure 9 603).

[0184] For example, when executing instructions stored in the memory 120 , the processor 110 may respectively use the cameras 141 , 142 , and 143 to obtain front images in front of the leading vehicle 11 .

[0185] Re-reference Figure 6A 、 Figure 6B and Figure 6C , images (e.g. Figure 6A The first image 601) may refer to an example of an image captured by the first camera 141, and the image (eg: Figure 6B The second image 602) may refer to an example of an image captured by the second camera 142, and the image (eg: Figure 6C The third image 603 may refer to an example of an image captured by the third camera 143. For example, the first camera 141 may have a first viewing angle, and the first image 601 may be based on the first viewing angle. The second camera 142 may have a second viewing angle narrower than the first viewing angle, and the second image 602 may be based on the second viewing angle. The third camera 143 may have a third viewing angle, and the third image 603 may be based on the third viewing angle. According to one embodiment, the processor 110 may obtain the first image 601 through the first camera 141, the second image 602 through the second camera 142, and the third image 603 through the third camera 143.

[0186] In action 1002 , the electronic device 100 may transmit at least a portion of the first image 601 , the second image 602 , and the third image 603 to the other electronic device 200 .

[0187] For example, when executing the instructions stored in the memory 120, the processor 110 may use the wireless communication device 130 to send at least a portion of the first image 601, the second image 602, and the third image 603 to the other electronic device 200. For example, the processor 110 may send the second image 602 to the first other electronic device 201 and send the third image 603 to the second other electronic device 202, but is not limited thereto.

[0188] According to one embodiment, the processor 110 may change the size of the front image to be transmitted and transmit it to reduce data capacity. When the resized front image is transmitted, the amount of processing by the wireless communication device 130 may be reduced. According to one embodiment, the processor 110 may change the size of the front image to a size input to the neural network model of the other electronic device 200. Since the other electronic device 200 does not require a high-resolution (e.g., 4K, FHD) image to recognize the external object 30 from the front image, the processor 110 may compress the front image into an image below HD for transmission.

[0189] In action 1003 , the electronic device 100 may obtain third information from the external object 30 included in the first image 601 .

[0190] For example, when executing instructions stored in the memory 120 , the processor 110 may identify the external object 30 included in at least a portion of the first image 601 , the second image 602 , and the third image 603 , and obtain third information regarding the external object 30 .

[0191] For example, the processor 110 may identify the external object 30 from the first image 601. The external object 30 may be identified using a neural network model. The neural network model may extract the external object 30 from the first image 601 and identify the external object 30 based on pre-learned images. For example, the third information may include information related to the type, location, size, and / or current state of the external object 30.

[0192] In action 1004 , the first other electronic device 201 may obtain fourth information related to the external object 30 from the external object 30 included in the second image 602 .

[0193] In action 1005 , the second other electronic device 202 may obtain fifth information related to the external object 30 from the external object 30 included in the third image 603 .

[0194] For example, the processor 210 of the first other electronic device 201 can obtain fourth information related to the external object 30 from the second image 602. The processor 210 of the second other electronic device 202 can obtain fifth information related to the external object 30 from the third image 603. The first other electronic device 201 and the second other electronic device 202 can recognize the external object 30 using a neural network model.

[0195] In action 1006 , the electronic device 100 may receive fourth information from the first other electronic device 201 and fifth information from the second other electronic device 202 .

[0196] For example, when executing instructions stored in the memory 120 , the processor 110 may receive fourth information from the first other electronic device 201 and fifth information from the second other electronic device 202 .

[0197] In action 1007 , the electronic device 100 may recognize the external object 30 based on the third information, the fourth information, and the fifth information.

[0198] For example, when executing instructions stored in the memory 120 , the processor 110 may identify the external object 30 based on third information acquired from the first image 601 , fourth information received from the first other electronic device 201 , and fifth information received from the second other electronic device 202 .

[0199] According to one embodiment, the electronic device 100 can reduce the amount of data processing, alleviate the data processing burden, and accurately identify external objects 30 by utilizing information provided by other electronic devices 200 (for example, the fourth information and the fifth information) to analyze images obtained through the first camera 141, the second camera 142, and the third camera 143.

[0200] Figure 11 FIG. 1 is a flowchart illustrating an operation of an electronic device recognizing an external object based on a main image and a sub-image according to an embodiment.

[0201] Figure 11 The actions described in the can be when the memory (for example: Figure 2 The instructions stored in the memory 120 and the memory 220) are executed by the processor (for example: Figure 2 When the processor 110 and the processor 210 of the electronic device (for example: Figure 2 The actions of the electronic device 100 and other electronic devices 200).

[0202] According to one embodiment, referring to Figure 8 The above description is substantially applicable to the electronic device 100 including multiple cameras 141, 142, 143. For example, the electronic device 100 may separate the first image 601, the second image 602, and the third image 603 acquired by the multiple cameras 141, 142, 143 into a main image and a sub-image for processing.

[0203] Reference Figure 11 In action 1101, the electronic device 100 may acquire a first image 601, a second image 602, and a third image 603. Action 1101 may correspond to Figure 10 Action 1001.

[0204] In action 1102 , the electronic device 100 may transmit at least a portion of the first image 601 , the second image 602 , and the third image 603 to the other electronic device 200 .

[0205] For example, when executing the instructions stored in the memory 120, the processor 110 may send at least a portion of the first image 601, the second image 602, and the third image 603 to the first other electronic device 201 and the second other electronic device 202. For example, the processor 110 may send the first image 601, the second image 602, and the third image 603 to both the first other electronic device 201 and the second other electronic device 202.

[0206] In action 1103 , the electronic device 100 may determine a first main image, a second main image, and a third main image.

[0207] For example, when executing instructions stored in the memory 120, the processor 110 may divide the front image (e.g., the first image 601, the second image 602, and the third image 603) into a first main image processed by the processor 110 based on a first FPS and a first sub-image processed by the processor 110 based on a second FPS. The first main image may refer to an image processed as a main image by the electronic device 100. The first sub-image is at least a portion of the remaining images other than the main image and may refer to an image processed as a sub-image by the electronic device 100. For example, when executing instructions stored in the memory 120, the processor 110 may divide the front image into a second main image processed by the first other electronic device 201 based on the first FPS and a second sub-image processed by the first other electronic device 201 based on the second FPS. For example, when executing instructions stored in the memory 120, the processor 110 may divide the front image into a third main image processed by the second other electronic device 202 based on the first FPS and a third sub-image processed by the second other electronic device 202 based on the second FPS. For example, the first main image may be the first image 601 , the second main image may be the second image 602 , and the third main image may be the third image 603 , but the present invention is not limited thereto.

[0208] In action 1104 , the electronic device 100 may obtain third information from the first main image (eg, the first image 601 ) and the first sub-image (eg, the second image 602 and the third image 603 ).

[0209] For example, when executing the instructions stored in the memory 120, the processor 110 processes the first main image based on the first FPS and processes the first sub-image based on the second FPS, thereby obtaining the image with the external object (e.g. Figure 1 For example, when the first image 601 is determined as the first primary image, the processor 110 processes the first image 601 based on the first FPS and processes the second image 602 and the third image 603 based on the second FPS, thereby obtaining the third information.

[0210] In action 1105 , the first other electronic device 201 may obtain fourth information from the second main image (eg, the second image 602 ) and the second sub-image (eg, the first image 601 and the third image 603 ).

[0211] For example, the first other electronic device 201 processes the second main image based on the first FPS and the second sub-image based on the second FPS, thereby obtaining the fourth information. For example, when the second image 602 is determined to be the second main image, the first other electronic device 201 processes the second image 602 based on the first FPS and processes the first image 601 and the third image 603 based on the second FPS, thereby obtaining the fourth information.

[0212] In action 1106 , the second other electronic device 202 may obtain fifth information from the third main image (eg, the third image 603 ) and the third sub-image (eg, the first image 601 and the second image 602 ).

[0213] For example, the second other electronic device 202 processes the third main image based on the first FPS and processes the third sub-image based on the second FPS, thereby obtaining the fifth information. For example, when the third image 603 is determined to be the third main image, the second other electronic device 202 processes the third image 603 based on the first FPS and processes the first image 601 and the second image 602 based on the second FPS, thereby obtaining the fifth information.

[0214] In action 1107 , the electronic device 100 may receive fourth information and fifth information.

[0215] For example, when executing the instructions stored in the memory 120 , the processor 110 may receive the fourth information acquired by the first other electronic device 201 and the fifth information acquired by the second other electronic device 202 .

[0216] In action 1108, the electronic device 100 may recognize the external object 30 based on the third information, the fourth information, and the fifth information. Action 1108 may correspond to Figure 8 For example, when executing the instructions stored in the memory 120, the processor 110 can identify the external object 30 located in front of the leading vehicle 11 by combining the third information, the fourth information, and the fifth information. The description of the action 805 can be substantially the same as that of the action 1108.

[0217] exist Figure 11 , the electronic device 100 is described as sending a first image 601, a second image 602, and a third image 603 to other electronic devices 200, and distinguishing the images as a main image and a sub-image, but is not limited thereto. For example, the electronic device 100 may also send the first image 601 and the second image 602 to a first other electronic device 201, and send the second image 602 and the third image 603 to a second other electronic device 202. The electronic device 100 may determine the second main image and the second sub-image, and determine the third main image and the third sub-image. For example, the electronic device 100 may determine the first image 601 as the second main image, and determine the third image 603 as the third main image. In this case, by distributing the processing of the images, the amount of data processing of the other electronic devices 200 may be reduced.

[0218] According to one embodiment, the electronic device 100 can distinguish the first image 601, the second image 602 and the third image 603 obtained by each of the multiple cameras 141, 142, and 143 into a main image and a sub-image, and use other electronic devices 200 to process part of the first image 601, the second image 602 and the third image 603, thereby reducing the data processing amount and accurately identifying the external object 30.

[0219] Figure 12 An example of a block diagram representing an autonomous driving system for a vehicle according to an embodiment is shown.

[0220] according to Figure 12 The autonomous driving system 1200 of a vehicle may be a deep learning network including a sensor 1203, an image preprocessor 1205, a deep learning network 1207, an artificial intelligence (AI) processor 1209, a vehicle control module 1211, a network interface 1213, and a communication unit 1215. In various embodiments, each element may be connected through various interfaces. For example, the sensor data sensed and output by the sensor 1203 may be fed to the image preprocessor 1205. The sensor data processed by the image preprocessor 1205 may be fed to the deep learning network 1207 running in the AI ​​processor 1209. The output of the deep learning network 1207 run by the AI ​​processor 1209 may be fed to the vehicle control module 1211. The intermediate results of the deep learning network 1207 run on the AI ​​processor 1209 may be fed to the AI ​​processor 1209. In various embodiments, the network interface 1213 is connected to the electronic devices in the vehicle (e.g., Figure 2 The electronic device 100 and / or other electronic device 200) communicates with the autonomous driving system 1200 to transmit autonomous driving path information and / or autonomous driving control commands for autonomous driving of the vehicle to the internal block components. In one embodiment, the network interface 1213 can be used to send sensor data acquired by one or more sensors 1203 to an external server. In some embodiments, the autonomous driving system 1200 may include more or fewer components as appropriate. For example, in some embodiments, the image preprocessor 1205 may be an optional component. As another example, a post-processing component (not shown) may be included in the autonomous driving system 1200 to perform post-processing on the output of the deep learning network 1207 before the output is provided to the vehicle control module 1211.

[0221] In some embodiments, sensor 1203 may include more than one sensor. In various embodiments, sensor 1203 may be attached to different locations on the vehicle. Sensor 1203 may face one or more different directions. For example, sensor 1203 may be attached to the front, sides, rear, and / or roof of the vehicle in various directions, such as forward-facing, rear-facing, and side-facing. In some embodiments, sensor 1203 may be an image sensor such as a high dynamic range camera. In some embodiments, sensor 1203 includes non-visual sensors. In some embodiments, in addition to an image sensor, sensor 1203 may also include radar (RADAR), laser radar (LiDAR), and / or ultrasonic sensors. In some embodiments, sensor 1203 is not mounted on a vehicle with vehicle control module 1211. For example, sensor 1203 may be included as part of a deep learning system for capturing sensor data and may be attached to the environment or road and / or mounted on surrounding vehicles.

[0222] In some embodiments, the image preprocessor 1205 can be used to preprocess sensor data from the sensor 1203. For example, the image preprocessor 1205 can be used to preprocess sensor data, to separate the sensor data into one or more constituent elements, and / or to post-process one or more constituent elements. In some embodiments, the image preprocessor 1205 can be a GPU, a CPU, an image signal processor, or a specialized image processor. In various embodiments, the image preprocessor 1205 can be a tone-mapper processor for processing high dynamic range data. In some embodiments, the image preprocessor 1205 can be a component of the AI ​​processor 1209.

[0223] In some embodiments, the deep learning network 1207 may be a deep learning network for implementing control commands for controlling an autonomous vehicle. For example, the deep learning network 1207 may be an artificial neural network such as a convolutional neural network (CNN) trained using sensor data, and the output of the deep learning network 1207 is provided to the vehicle control module 1211.

[0224] In some embodiments, the AI ​​processor 1209 may be a hardware processor for running the deep learning network 1207. In some embodiments, the AI ​​processor 1209 is a specialized AI processor for performing inference on sensor data through a convolutional neural network. In some embodiments, the AI ​​processor 1209 may be optimized for the bit depth of the sensor data. In some embodiments, the AI ​​processor 1209 may be optimized for deep learning calculations such as calculations of neural networks including convolution, dot product, vector and / or matrix calculations. In some embodiments, the AI ​​processor 1209 may be implemented by multiple graphics processing units (GPUs) capable of efficiently performing parallel processing.

[0225] In various embodiments, the AI ​​processor 1209 performs deep learning analysis on sensor data received from one or more sensors 1203 during operation of the AI ​​processor 1209 and can be coupled via an input / output interface to a memory configured to provide the AI ​​processor with instructions for triggering the determination of machine learning results for at least partially autonomous operation of the vehicle. In some embodiments, the vehicle control module 1211 processes commands output from the AI ​​processor 1209 for controlling the vehicle and can be configured to translate the output of the AI ​​processor 1209 into instructions for controlling various modules of the vehicle. In some embodiments, the vehicle control module 1211 is configured to control the vehicle for autonomous driving. In some embodiments, the vehicle control module 1211 can adjust the steering and / or speed of the vehicle. For example, the vehicle control module 1211 can be configured to control vehicle movement, such as deceleration, acceleration, steering, lane changes, and lane keeping. In some embodiments, the vehicle control module 1211 can generate control signals for controlling vehicle lighting, such as brake lights, turn signals, and headlights. In some embodiments, the vehicle control module 1211 can be used to control vehicle audio-related systems, such as the vehicle's sound system, vehicle's audio warnings, vehicle's microphone system, vehicle's horn system, etc.

[0226] In some embodiments, the vehicle control module 1211 is used to control notification systems, including those for notifying passengers and / or the driver of travel events, such as approaching a predetermined destination or a potential collision. In some embodiments, the vehicle control module 1211 can be used to adjust sensors, such as the vehicle's sensors 1203. For example, the vehicle control module 1211 can modify the orientation of the sensors 1203, change the output resolution and / or format type of the sensors 1203, increase or decrease the capture rate, adjust the dynamic range, or adjust the focus of the camera. In addition, the vehicle control module 1211 can turn sensors on and off individually or collectively.

[0227] In some embodiments, the vehicle control module 1211 can be used to change the parameters of the image preprocessor 1205 by modifying the frequency range of the filter, or by adjusting the edge detection parameters for feature and / or object detection, or by adjusting the bit depth and channels. In various embodiments, the vehicle control module 1211 can be used to control the autonomous driving and / or driver assistance functions of the vehicle.

[0228] In some embodiments, the network interface 1213 may function as an internal interface between the components of the autonomous driving system 1200 and the communication unit 1215. Specifically, the network interface 1213 may be a communication interface for receiving and / or transmitting data, including voice data. In various embodiments, the network interface 1213 may be connected to an external server to connect a voice call through the communication unit 1215, or to receive and / or transmit text messages, or to transmit sensor data, or to update the vehicle's software through the autonomous driving system, or to update the vehicle's autonomous driving system software.

[0229] In various embodiments, the communication unit 1215 may include various wireless interfaces such as cellular or WiFi. For example, the network interface 1213 may be used to receive updates on operating parameters and / or instructions for the sensor 1203, image preprocessor 1205, deep learning network 1207, AI processor 1209, and vehicle control module 1211 from an external server connected via the communication unit 1215. For example, the machine learning model of the deep learning network 1207 may be updated using the communication unit 1215. According to another example, the communication unit 1215 may be used to update operating parameters of the image preprocessor 1205, such as image processing parameters, and / or the firmware of the sensor 1203.

[0230] In another embodiment, the communication unit 1215 can be used to activate communications for emergency services and emergency contacts in the event of an accident or near-accident. For example, in the event of a collision, the communication unit 1215 can be used to call emergency services for assistance and can be used to notify emergency services of the details of the collision and the location of the vehicle. In various embodiments, the communication unit 1215 can update or obtain an estimated time of arrival and / or destination location.

[0231] According to one embodiment, Figure 12 The illustrated autonomous driving system 1200 may also be comprised of the vehicle's electronic device 100. According to one embodiment, when a user deactivates the autonomous driving function while the vehicle is operating autonomously, the AI ​​processor 1209 of the autonomous driving system 1200 controls the deep learning network to learn the vehicle's autonomous driving software by inputting information related to the autonomous driving deactivation event as training data.

[0232] Figure 13 and Figure 14 An example of a block diagram showing an autonomous driving mobile object according to an embodiment is shown. Figure 15 An example of a gateway associated with a user device according to various embodiments is shown.

[0233] Reference Figure 13 According to this embodiment, the autonomous driving mobile body 1300 may include: a control device 1400, sensing modules 1304a, 1304b, 1304c, 1304d, an engine 1306 and a user interface (UI) 1308.

[0234] The autonomous vehicle 1300 may have an autonomous driving mode or a manual mode. For example, the vehicle may be switched from the manual mode to the autonomous driving mode or vice versa based on user input received through the user interface 1308 .

[0235] When the autonomous driving mobile object 1300 operates in the autonomous driving mode, the autonomous driving mobile object 1300 may operate under the control of the control device 1400 .

[0236] In this embodiment, the control device 1400 may include a controller 1420 including a memory 1422 and a processor 1424 , a sensor 1410 , a wireless communication device 1430 , and an object detection device 1440 .

[0237] The object detection device 1440 may perform all or part of the functions of the distance measurement device.

[0238] That is, in this embodiment, the object detection device 1440 is a device for detecting objects located outside the autonomous driving mobile body 1300. The object detection device 1440 can detect objects located outside the autonomous driving mobile body 1300 and generate object information corresponding to the detection result.

[0239] The object information may include information on the presence or absence of the object, position information of the object, distance information between the moving body and the object, and relative speed information between the moving body and the object.

[0240] Objects may include lane markings, other vehicles, pedestrians, traffic signals, lights, roads, structures, speed bumps, terrain features, animals, and other objects located outside of the autonomous vehicle 1300. Traffic signals may include traffic lights, traffic signs, and patterns or text drawn on the road surface. Light may be generated by lights installed in other vehicles, streetlights, or sunlight.

[0241] Additionally, structures can be objects located around the road and fixed to the ground. For example, structures can include streetlights, roadside trees, buildings, utility poles, traffic lights, and bridges. Terrain can include mountains and hills.

[0242] Such an object detection device 1440 may include a camera module. The controller 1420 may extract object information from an external image captured by the camera module and cause the controller 1420 to process information related thereto.

[0243] In addition, the object detection device 1440 may also include an imaging device for identifying the external environment. In addition to LIDAR, radar, GPS devices, distance measurement devices (odometry), other computer vision devices, ultrasonic sensors, and infrared sensors may also be used. These devices can be selected or activated simultaneously as needed to achieve more accurate sensing.

[0244] On the other hand, the distance measurement device according to an embodiment of the present invention calculates the distance between the autonomous driving mobile body 1300 and an object, is connected to the control device 1400 of the autonomous driving mobile body 1300, and controls the movement of the mobile body based on the calculated distance.

[0245] For example, when there is a possibility of collision between the autonomous driving vehicle 1300 and an object based on the distance between the autonomous driving vehicle 1300 and the object, the autonomous driving vehicle 1300 may control the brakes to slow down or stop the vehicle. For another example, when the object is moving, the autonomous driving vehicle 1300 may control its speed to maintain a predetermined distance from the object.

[0246] The distance measurement device according to an embodiment of the present invention may be configured as a module within the control device 1400 of the autonomous driving mobile object 1300. That is, the memory 1422 and the processor 1424 of the control device 1400 may implement the collision avoidance method according to the present invention in a software manner.

[0247] In addition, the sensor 1410 can obtain various sensing information by connecting the internal / external environment of the mobile object to the sensing modules 1304a, 1304b, 1304c, and 1304d. The sensor 1410 may include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor, a collision sensor, a wheel sensor), a speed sensor, a tilt sensor, a weight sensing sensor, a heading sensor, a gyro sensor, a position module, a mobile forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a mobile internal temperature sensor, a mobile internal humidity sensor, an ultrasonic sensor, an illumination sensor, an accelerator pedal position sensor, a brake pedal position sensor, and the like.

[0248] Thus, the sensor 1410 can obtain sensing signals for mobile body posture information, mobile body collision information, mobile body direction information, mobile body position information (GPS information), mobile body angle information, mobile body speed information, mobile body acceleration information, mobile body slope information, mobile body forward / backward information, battery information, fuel information, tire information, mobile body light information, mobile body internal temperature information, mobile body internal humidity information, steering wheel rotation angle, mobile body external illuminance, pressure applied to the accelerator pedal, pressure applied to the brake pedal, etc.

[0249] In addition, in addition to this, sensor 1410 may also include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor, an intake air temperature sensor, a water temperature sensor, a throttle position sensor, a top dead center sensor, a crank angle sensor, etc.

[0250] As described above, the sensor 1410 may generate moving body state information based on the sensing data.

[0251] The wireless communication device 1430 is configured to implement wireless communication between the autonomous driving mobile bodies 1300. For example, the autonomous driving mobile body 1300 can communicate with the user's mobile phone, or other wireless communication devices 1430, other mobile bodies, central devices (traffic control devices), servers, etc. The wireless communication device 1430 can send and receive wireless signals according to the connection wireless protocol. The wireless communication protocol can be Wi-Fi, Bluetooth, Long-Term Evolution (LTE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Global Systems for Mobile Communications (GSM), but the communication protocol is not limited thereto.

[0252] In addition, in this embodiment, the autonomous driving mobile body 1300 can also realize communication between mobile bodies through the wireless communication device 1430. That is, the wireless communication device 1430 can communicate with other mobile bodies on the road through vehicle-to-vehicle (V2V) communication. The autonomous driving mobile body 1300 can send and receive information such as driving warnings and traffic information through inter-vehicle communication, and can also request information from other mobile bodies or receive requests from other mobile bodies. For example, the wireless communication device 1430 can perform V2V communication through a dedicated short-range communication (DSRC) device or a cellular V2V (Cellular-V2V, C-V2V) device. In addition, in addition to communication between vehicles, communication between the vehicle and other objects (such as electronic devices carried by pedestrians, etc.) can also be realized through the wireless communication device 1430.

[0253] In addition, the wireless communication device 1430 can obtain information generated from infrastructure located on the road (traffic lights, CCTV, road side units (RSU), evolved Node B (eNode B), etc.) or various mobile bodies (Mobility) including other autonomous driving (Autonomous Driving) / non-autonomous driving (Non-Autonomous Driving) vehicles through a non-terrestrial network (Non-Terrestrial Network) that is not a terrestrial network (Terrestrial Network), and use the generated information as information for performing autonomous driving of the autonomous driving mobile body 1300.

[0254] For example, the wireless communication device 1430 can wirelessly communicate with the low Earth orbit (LEO) satellite system, medium Earth orbit (MEO) satellite system, geostationary orbit (GEO) satellite system, high altitude platform (HAP) system, etc. that constitute the non-ground network through a non-ground network dedicated antenna installed on the autonomous driving mobile body 1300.

[0255] For example, the wireless communication device 1430 can perform wireless communications with various platforms constituting the 5th Generation New Radio Non-Terrestrial Network (5GNR NTN) according to a wireless access standard corresponding to the 5th Generation New Radio Non-Terrestrial Network (5GNR NTN) standard specification currently discussed in 3GPP, but is not limited thereto.

[0256] In this embodiment, the controller 1420 can consider various information such as the location, current time, available power, etc. of the autonomous driving mobile body 1300, select a platform that can appropriately perform NTN communication, and control the wireless communication device 1430 to perform wireless communication with the selected platform.

[0257] In this embodiment, the controller 1420 is a unit that controls the overall operation of each unit within the autonomous driving mobile body 1300. It can be configured by the manufacturer of the mobile body during manufacturing or can be additionally configured after manufacturing to perform the autonomous driving function. Alternatively, the controller 1420 configured at the time of manufacturing can be upgraded to include a configuration for continuously performing additional functions. The controller 1420 can also be called an electronic control unit (ECU).

[0258] The controller 1420 collects various data from the connected sensors 1410, object detection device 1440, wireless communication device 1430, and the like, and transmits control signals based on the collected data to the sensors 1410, engine 1306, user interface 1308, wireless communication device 1430, and object detection device 1440 included in other components of the mobile body. Although not shown, the control signals may also be sent to a throttle device, a braking system, a steering device, or a navigation device related to the travel of the mobile body.

[0259] In this embodiment, the controller 1420 can control the engine 1306. For example, the engine 1306 can be controlled to sense the speed limit of the road on which the autonomous driving mobile body 1300 is traveling and control the driving speed not to exceed the speed limit, or the engine 1306 can be controlled to increase the driving speed of the autonomous driving mobile body 1300 within a range that does not exceed the speed limit.

[0260] Furthermore, if the autonomous vehicle 1300 approaches or deviates from a lane line during its travel, the controller 1420 determines whether this approach or deviation is due to a normal driving situation or another driving situation. Based on the determination, the controller 1420 controls the vehicle's travel by controlling the engine 1306. Specifically, the autonomous vehicle 1300 can detect lane lines formed on both sides of the lane in which the vehicle is traveling. In this case, the controller 1420 determines whether the autonomous vehicle 1300 approaches or deviates from a lane line. If the autonomous vehicle 1300 approaches or deviates from a lane line, the controller 1420 determines whether the movement is due to a normal driving situation or another driving situation. An example of a normal driving situation is a situation where the vehicle needs to change lanes. An example of an other driving situation is a situation where the vehicle does not need to change lanes. If the controller 1420 determines that the autonomous driving mobile body 1300 approaches the lane line or deviates from the lane line without changing the lane of the mobile body, the driving of the autonomous driving mobile body 1300 can be controlled so that the autonomous driving mobile body 1300 does not deviate from the lane line and drives normally in the lane.

[0261] If there are other moving objects or obstacles in front of the moving object, the engine 1306 or the braking system can be controlled to slow the moving object. In addition to speed, the trajectory, driving path, and steering angle can also be controlled. Alternatively, the controller 1420 can generate the necessary control signals based on the recognition of other external environmental information such as the moving object's lane markings, driving signals, etc. to control the movement of the moving object.

[0262] In addition to generating its own control signals, the controller 1420 can also communicate with surrounding mobile objects or a central server, and send commands for controlling peripheral devices through the received information, thereby controlling the travel of the mobile object.

[0263] Furthermore, accurate recognition of a moving object or lane marking according to this embodiment may be difficult when the position or viewing angle of camera module 1450 changes. To prevent this, controller 1420 may also generate a control signal to calibrate camera module 1450. Therefore, in this embodiment, controller 1420 generates a calibration control signal to camera module 1450 to maintain its proper installation position, orientation, and viewing angle, even if the camera module 1450's installation position changes due to vibration or impact generated by the movement of autonomous vehicle 1300. When the difference between pre-stored initial camera module installation position, orientation, and viewing angle information and the initial camera module installation position, orientation, and viewing angle information measured during the driving of autonomous vehicle 1300 exceeds a threshold, controller 1420 may generate a control signal to calibrate the camera module.

[0264] In this embodiment, the controller 1420 may include a memory 1422 and a processor 1424. The processor 1424 may execute software stored in the memory 1422 according to control signals from the controller 1420. Specifically, the controller 1420 stores data and commands for executing the lane detection method according to the present invention in the memory 1422. The commands may be executed by the processor 1424 to implement one or more methods disclosed herein.

[0265] In this case, the memory 1422 may be stored in a non-volatile recording medium that can be executed by the processor 1424. The memory 1422 may store software and data through appropriate internal or external devices. The memory 1422 may be composed of a memory 1422 device connected to RAM, ROM, a hard disk, and an adapter (dongle).

[0266] The memory 1422 may store at least an operating system (OS), user applications, and executable commands. The memory 1422 may also store application data and arrange data structures.

[0267] Processor 1424 may be a microprocessor or a suitable electronic processor, such as a controller, microcontroller, or state machine.

[0268] Processor 1424 may be implemented as a combination of computing devices, and the computing device may be a digital signal processor, a microprocessor, or a suitable combination thereof.

[0269] On the other hand, the autonomous driving vehicle 1300 may also include a user interface 1308 for providing user input to the control device 1400. The user interface 1308 may allow the user to input information through appropriate interaction. For example, it may be implemented through a touch screen, a keyboard, action buttons, etc. The user interface 1308 transmits the input or command to the controller 1420, and the controller 1420 may execute the control action of the vehicle in response to the input or command.

[0270] In addition, the user interface 1308 may enable the autonomous vehicle 1300 to communicate with a device external to the autonomous vehicle 1300 via the wireless communication device 1430. For example, the user interface 1308 may be linked to a mobile phone, tablet computer, or other computer device.

[0271] Furthermore, in this embodiment, the autonomous vehicle 1300 is described as including an engine 1306, but may also include other types of propulsion systems. For example, the vehicle may be powered by electricity, hydrogen, or a hybrid system combining these. Therefore, the controller 1420 may include propulsion mechanisms corresponding to the propulsion system of the autonomous vehicle 1300 and provide control signals corresponding to each propulsion mechanism.

[0272] In the following, reference will be made to Figure 14 The detailed configuration of the control device 1400 according to this embodiment will be described in more detail.

[0273] The control device 1400 includes a processor 1424. The processor 1424 can be a general-purpose single-chip or multi-chip microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, etc. The processor can also be referred to as a CPU. In addition, in this embodiment, the processor 1424 can be used as a combination of multiple processors.

[0274] Furthermore, the control device 1400 further includes a memory 1422. The memory 1422 may also be any electronic component capable of storing electronic information. In addition to a single memory, the memory 1422 may also include a combination of memories 1422.

[0275] Data and instructions 1422a for executing the distance measurement method of the distance measurement device according to the present invention may also be stored in the memory 1422. When the processor 1424 executes the instructions 1422a, all or part of the instructions 1422a and the data 1422b required for executing the command may also be loaded 1424a, 1424b onto the processor 1424.

[0276] The control device 1400 may also include a transmitter 1430a, a receiver 1430b, or a transceiver 1430c to allow transmission and reception of signals. One or more antennas 1432a, 1432b may also be electrically connected to the transmitter 1430a, the receiver 1430b, or each transceiver 1430c, and additional antennas may also be included.

[0277] The control device 1400 may further include a digital signal processor (DSP) 1470. Through the DSP 1470, the mobile object can quickly process digital signals.

[0278] The control device 1400 may also include a communication interface 1480. The communication interface 1480 may also include one or more ports and / or communication modules for connecting other devices to the control device 1400. The communication interface 1480 may allow a user to interact with the control device 1400.

[0279] The various components of the control device 1400 may also be connected together via one or more buses 1490 , which may also include a power bus, a control signal bus, a status signal bus, a data bus, etc. Under the control of the processor 1424 , the various components may communicate information with each other via the bus 1490 and perform desired functions.

[0280] On the other hand, in various embodiments, the control device 1400 can be associated with a gateway to communicate with the secure cloud. Figure 15 , the control device 1400 may be associated with a gateway 1505 for providing information acquired from at least one of the components (1501 to 1504) of the vehicle 1500 to a security cloud 1506. For example, the gateway 1505 may be included in the control device 1400. As another example, the gateway 1505 may also be configured as an additional device within the vehicle 1500 that is distinct from the control device 1400. The gateway 1505 communicatively connects the software management cloud 1509, the security cloud 1506, and the network within the vehicle 1500 protected by the in-vehicle security software 1510, which have different networks.

[0281] For example, component 1501 may be a sensor. For example, the sensor may be used to obtain information related to at least one of the state of vehicle 1500 and the state of the surrounding area of ​​vehicle 1500. For example, component 1501 may include a sensor.

[0282] For example, component 1502 may be an ECU, which may be used for engine control, transmission control, airbag control, and tire pressure management.

[0283] For example, component 1503 may be an instrument cluster. For example, the instrument cluster may represent a panel located in front of the driver's seat in a dashboard. For example, the instrument cluster may be configured to display information necessary for driving to the driver (or passenger). For example, the instrument cluster may be configured to display at least one of a visual element indicating the engine's revolutions per minute (RPM), a visual element indicating the speed of vehicle 1500, a visual element indicating the remaining fuel level, a visual element indicating a gear position, and a visual element indicating information acquired through component 1501.

[0284] For example, component 1504 may be an in-vehicle communication (telematics) device. For example, the in-vehicle communication device may represent a device that provides various mobile communication services (e.g., location information, safe driving, etc.) within vehicle 1500 by combining wireless communication technology and GPS technology. For example, the in-vehicle communication device may be used to connect vehicle 1500 to a driver, a cloud (e.g., a safety cloud 1506), and / or a surrounding environment. For example, the in-vehicle communication device may be configured to support high bandwidth and low latency for technologies specified in 5G NR (e.g., 5G NR's V2X technology, 5G NR's NTN technology). For example, the in-vehicle communication device may be configured to support autonomous driving of vehicle 1500.

[0285] For example, gateway 1505 can be used to connect the network within vehicle 1500 with a software management cloud 1509 and a security cloud 1506, which are external networks. For example, software management cloud 1509 can be used to update or manage at least one software required for driving and managing vehicle 1500. For example, software management cloud 1509 can be linked with in-car security software 1510 installed in the vehicle. For example, in-car security software 1510 can be used to provide security functions within vehicle 1500. For example, in-car security software 1510 can use an encryption key obtained from an external authorized server to encrypt data sent and received over the in-vehicle network to encrypt the in-vehicle network. In various embodiments, the encryption key used by in-vehicle security software 1510 can be generated in accordance with vehicle identification information (such as the license plate or vehicle identification number (VIN)) or information uniquely assigned to each user (e.g., user identification information).

[0286] In various embodiments, gateway 1505 can transmit data encrypted by in-vehicle security software 1510 to software management cloud 1509 and / or security cloud 1506 based on the encryption key. Software management cloud 1509 and / or security cloud 1506 decrypt the data encrypted by the in-vehicle security software 1510's encryption key using a decryption key, thereby identifying the vehicle or user from which the data was received. For example, the decryption key is a unique key corresponding to the encryption key, so software management cloud 1509 and / or security cloud 1506 can identify the data sender (e.g., the vehicle or user) based on the data decrypted by the decryption key.

[0287] For example, gateway 1505 is configured to support in-vehicle security software 1510 and may be associated with control device 1400. For example, gateway 1505 may be associated with control device 1400 to support a connection between a client device 1507 connected to a secure cloud 1506 and control device 1400. As another example, gateway 1505 may be associated with control device 1400 to support a connection between a third-party cloud 1508 connected to secure cloud 1506 and control device 1400. However, this is not limiting.

[0288] In various embodiments, gateway 1505 can be used to connect a software management cloud 1509 for managing the operating software of vehicle 1500 with vehicle 1500. For example, software management cloud 1509 can monitor whether the operating software of vehicle 1500 needs to be updated, and when it is detected that the operating software of vehicle 1500 needs to be updated, provide data for updating the operating software of vehicle 1500 through gateway 1505. As another example, software management cloud 1509 can receive a user request to update the operating software of vehicle 1500 from vehicle 1500 through gateway 1505, and provide data for updating the operating software of vehicle 1500 based on the received user request. However, the present invention is not limited to this.

[0289] Figure 16 is a diagram for explaining the operation of an electronic device for training a neural network based on a learning data set according to one embodiment.

[0290] Reference Figure 16 The described actions may be performed by the above electronic devices (for example: Figure 2 Executed by the electronic device 100).

[0291] Reference Figure 16In action 1602, an electronic device according to an embodiment may obtain a learning data set. The electronic device may obtain a learning data set for supervised learning. The learning data may include input data and ground truth data pairs corresponding to the input data. The ground truth data may represent output data to be obtained from a neural network that receives the input data as the ground truth data pairs. The ground truth data may be obtained by the electronic device.

[0292] For example, when training a neural network to recognize an image, the learning data may include information related to the image and one or more subjects included in the image. The information may include a category (category or class) of a subject that can be identified by the image. The information may include the position, width, height and / or size of a visual object corresponding to the subject within the image. The learning data set identified by action 1602 may include multiple learning data pairs. In the example of training a neural network to recognize an image, the learning data set identified by the electronic device may include multiple images and real data corresponding to each of the multiple images.

[0293] Reference Figure 16 In action 1604, the electronic device according to an embodiment may train a neural network based on a learning data set. In an embodiment of training a neural network based on supervised learning, the electronic device may input input data included in the learning data into an input layer of the neural network. Figure 17 An example of a neural network including the input layer is described. The electronic device can obtain output data of the neural network corresponding to the input data from the output layer of the neural network that receives the input data through the input layer.

[0294] In one embodiment, the training of action 1604 may be performed based on the difference between the output data and the real data included in the learning data and corresponding to the input data. For example, the electronic device may adjust one or more parameters related to the neural network based on a gradient descent algorithm (e.g., see later). Figure 17 The electronic device may adjust the one or more parameters to reduce the difference. The electronic device may adjust the one or more parameters to reduce the difference. The electronic device may adjust the one or more parameters to reduce the difference. The electronic device may perform the tuning of the neural network based on the output data using a function defined to evaluate the performance of the neural network, such as a cost function. The difference between the output data and the real data may be included as an example of the cost function.

[0295] Reference Figure 16In action 1606, the electronic device according to one embodiment may identify whether valid output data is output from the neural network trained in action 1604. Valid output data may mean that the difference (or cost function) between the output data and the true data satisfies the conditions set for using the neural network. For example, when the average value and / or the maximum value of the difference between the output data and the true data is below a specified threshold, the electronic device may determine that valid output data is output from the neural network.

[0296] When no valid output data is output from the neural network (action 1606 is No), the electronic device may repeatedly perform training of the neural network based on action 1604. The embodiment is not limited thereto, and the electronic device may repeatedly perform actions 1602 and 1604.

[0297] When valid output data is obtained from the neural network (yes in action 1606), the electronic device according to one embodiment may use the trained neural network according to action 1608. For example, the electronic device may input other input data, which is different from the input data input to the neural network, into the neural network as learning data. The electronic device may use the output data obtained from the neural network that received the other input data as a result of reasoning performed on the other input data by the neural network.

[0298] Figure 17 is a block diagram of an electronic device according to an embodiment.

[0299] Figure 17 The electronic device 100 may include the above-mentioned electronic devices.

[0300] For example, refer to Figure 16 The action described can be Figure 17 The electronic device 100 and / or Figure 17 Executed by processor 1710.

[0301] Reference Figure 17 , the processor 1710 of the electronic device 100 may perform calculations related to the neural network 1730 stored in the memory 1720. The processor 1710 may include at least one of a CPU, a GPU, and an NPU. The NPU may be implemented as a chip separate from the CPU, or may be integrated into a chip such as a CPU in the form of a system on a chip. The NPU integrated into the CPU may be referred to as a neural core and / or an AI accelerator.

[0302] Reference Figure 17, the processor 1710 may identify a neural network 1730 stored in the memory 1720. The neural network 1730 may include a combination of an input layer 1732, one or more hidden layers 1734 (or intermediate layers), and an output layer 1736. The layers described above (e.g., the input layer 1732, one or more hidden layers 1734, and the output layer 1736) may include multiple nodes. The number of hidden layers 1734 may vary depending on the embodiment, and a neural network 1730 including multiple hidden layers 1734 may be referred to as a deep neural network. The action of training the deep neural network may be referred to as deep learning.

[0303] In one embodiment, when neural network 1730 has a feedforward neural network structure, a first node included in a specific layer may be connected to all second nodes included in other layers before the specific layer. Parameters stored for neural network 1730 in memory 1720 may include weights assigned to connections between the second nodes and the first node. In neural network 1730 having a feedforward neural network structure, the value of the first node may correspond to a weighted sum of the values ​​assigned to the second node based on the weights assigned to the connections connecting the second node and the first node.

[0304] In one embodiment, when neural network 1730 has a convolutional neural network structure, a first node included in a specific layer may correspond to a weighted sum of a portion of second nodes included in other layers prior to the specific layer. The portion of second nodes corresponding to the first node in the second nodes may be identified by a filter corresponding to the specific layer. Parameters stored in memory 1720 for neural network 1730 may include weighted values ​​representing the filter. The filter may include one or more nodes in the second nodes to be used to calculate the weighted sum of the first node, and a weighted value corresponding to each of the one or more nodes.

[0305] According to an embodiment, the processor 1710 of the electronic device 100 can train the neural network 1730 using the learning data set 1740 stored in the memory 1720. Based on the learning data set 1740, the processor 1710 can perform a reference Figure 16 The described actions adjust one or more parameters stored in memory 1720 for neural network 1730.

[0306] According to one embodiment, the processor 1710 of the electronic device 100 can perform object detection, object recognition, and / or object classification using a neural network 1730 trained based on a learning data set 1740. The processor 1710 can input an image (or video) acquired through the camera 1750 into the input layer 1732 of the neural network 1730. Based on the input layer 1732 to which the image is input, the processor 1710 can sequentially acquire the values ​​of the nodes of the layers included in the neural network 1730 and acquire a set of values ​​(e.g., output data) of the nodes of the output layer 1736. The output data can be used as a result of reasoning about the information included in the image using the neural network 1730. The embodiment is not limited thereto, and the processor 1710 can input an image (or video) acquired from an external electronic device connected to the electronic device 100 through the communication circuit 1760 into the neural network 1730.

[0307] In one embodiment, the neural network 1730 trained to process an image can be used to identify (object detection) an area corresponding to a subject within the image, and / or identify (object recognition and / or object classification) the category of the subject presented in the image. For example, the electronic device 100 can use the neural network 1730 to segment the area corresponding to the subject within the image based on the shape of a rectangle such as a bounding box. For example, the electronic device 100 can use the neural network 1730 to identify at least one category matching the subject from a plurality of specified categories.

[0308] Figure 18A and Figure 18B An example of a vehicle is shown.

[0309] The vehicles performing the convoy travel mentioned above may be conventional trucks.

[0310] Over the years, the trucking industry has experienced sustained growth and expanded its service offerings to address more complex supply chains. These services include last-mile deliveries, drop-trailer programs, and intermodal transportation at ports (where freight is transported to its destination using two or more different modes of transportation, such as ship and rail, or ship and air).

[0311] As such, there are many ways to transport cargo, and manufacturers of cargo transportation-related equipment have designed different types of equipment for transporting cargo according to different transportation needs.

[0312] In this specification, trucks whose main purpose is to tow freight are generally referred to as tractors.

[0313] The tractors described in this specification can be divided into conventional trucks (or bonneted trucks), cab-over trucks (or cab-over engines), and semi-conventional trucks that are between conventional trucks and cab-over trucks, depending on the position and shape of the tractor cab.

[0314] Conventional trucks have an engine and a hood located above the front axle in front of a tractor cab, and a driver sitting behind the front axle. The tractor engine is located in front of the driver, and is a shape of tractor mainly used in North America.

[0315] On the other hand, a cab-over truck has a structure where the driver sits in front of the front axle, with the cab of the tractor located at the front end of the tractor. This is a so-called "flat face" or "flat nose" tractor with a flat front surface. The tractor's engine is located below the driver, and this is a tractor shape primarily used in most countries in Europe and Asia.

[0316] Just as there are many types of tractors depending on their purpose and needs, there are also many types of trailers towed by tractors. The most representative types of trailers are full-trailers and semi-trailers. Full-trailers and semi-trailers can be distinguished by whether the trailer is equipped with both a front axle and a rear axle. Such trailers can be connected to box trucks or tractors via a coupling device.

[0317] Specifically, a full trailer is a commercial freight trailer equipped with both front and rear axles. Designed so that its total load is supported solely by the trailer, it is capable of fully supporting its own weight without relying on a tractor, and is equipped with a drawbar for coupling to a hauling unit or towing unit such as a tractor. It is primarily used in the United States, Canada, and elsewhere.

[0318] On the other hand, a semi-trailer is a freight trailer that is only equipped with a rear axle but not a front axle, and a large part of the load can be supported by a tractor connected by a hook (hitch) that is called a "fifth wheel (fifth wheel, or steering wheel)". When the semi-trailer is separated from the tractor and is in a stationary state, the landing gear (landing gear) installed at the bottom of the semi-trailer can be vertically unfolded to the ground to support the load of the trailer. The combination of a semi-trailer and a tractor is called a semi-trailer truck (semi-trailer truck) (also referred to as "semi-trailer", "tractor-trailer (tractor-trailer)", "semi-truck (semi-truck)", "big truck (Big rig)" or "semi-trailer (semi)") in the United States. The above-mentioned "steering wheel (fifth wheel (fifth wheel))" refers to a horizontal wheel attached to the tractor axle of the trailer truck for facilitating the direction change of the trailer, also known as a "fifth wheel". A fifth wheel is a device that movably couples a tractor and a semi-trailer, and generally includes a lower portion consisting of a kingpin mounted on the semi-trailer, a trunnion plate securely fixed to the tractor, and a locking device.

[0319] In the following description, based on the above-mentioned tractor / trailer terminology, for convenience of explanation, "trailer" will refer to a cargo transport vehicle connected to a tractor for a trailer, and "tractor" will refer to a towing vehicle used to move the trailer. Furthermore, in order to maximize the elimination of rights limitations based on the embodiments described in the detailed description, in the present invention, the tractor hauling / towing a "trailer" may be used interchangeably with the "towing vehicle," and the trailer towed by the tractor may be used interchangeably with the "towed vehicle."

[0320] In addition, for the sake of convenience, preferably, the “trailer” described throughout this specification should be understood to refer to a “semi-trailer”, but is not limited thereto.

[0321] This manual Figure 18A and Figure 18B The trailer 1820 shown is shown as a "semi-trailer" form, but this is for the convenience of explanation and it should not be understood that the embodiments of the present invention are only applicable to the "semi-trailer" form.

[0322] Reference Figure 18A and Figure 18B , a vehicle 1800 including a tractor or tractor unit 1810 and a semi-trailer 1820 is exemplarily shown. Figure 18A The tractor 1810 and the semi-trailer 1820 are shown as being disconnected. Figure 18BThe state of the tractor 1810 and the semi-trailer 1820 being connected is shown. In one embodiment, the semi-trailer 1820 can be selectively connected via a steering wheel hook 1860 carried by the tractor 1810, and the steering wheel hook 1860 can be fastened to a king pin 1880 fixed to the semi-trailer 1820 in a known manner. The vehicle 1800 including the tractor 1810 and the semi-trailer 1820 can be called a truck. The vehicle 1800 can include only the tractor 1810. Figure 18A and Figure 18B The semi-trailer 1820 shown is shown as a "semi-trailer" form, but this is only for the convenience of explanation and should not be understood as the embodiment of the present disclosure is only applicable to the "semi-trailer" form. Figure 18A and Figure 18B The tractor 1810 shown is shown as a "cab-type truck" form, but this is only for the convenience of explanation and it should not be understood that the embodiments of the present disclosure are only applicable to the "cab-type truck" form.

[0323] In one embodiment, a tractor 1810 may include a forward portion 1811 and a rear portion 1812. The forward portion 1811 may include a cab (or cabin) in which a driver sits. The rear portion 1812 may be configured with a steering wheel hook 1860 for coupling to a semi-trailer 1820. In one embodiment, the semi-trailer 1820 may include a kingpin 1880 coupled to the steering wheel hook 1860 of the tractor 1810 and an undercarriage 1890 for supporting the semi-trailer 1820 from the ground when the semi-trailer 1820 is not coupled to the tractor 1810. The kingpin 1880 and the undercarriage 1890 may be disposed at the bottom of the semi-trailer 1820.

[0324] In one embodiment, the tractor 1810 may include an internal combustion engine, a motor, or a combination thereof, referred to as an engine. The tractor 1810 may include a battery and / or a fuel tank (e.g., a fuel tank designed to store gasoline, diesel, liquefied natural gas (LNG), liquefied petroleum gas (LPG), and / or hydrogen). For example, a tractor 1810 that includes a rechargeable battery and a motor driven by the electrical energy stored in the battery may be referred to as an electric vehicle (EV) and / or an electric truck. For example, a tractor 1810 that includes not only a battery and a motor but also a fuel tank and an engine may be referred to as a hybrid vehicle (e.g., a plug-in hybrid electric vehicle (PHEV)).

[0325] In one embodiment, the semi-trailer 1820 can be coupled to or detached from the tractor 1810. For example, the semi-trailer 1820 can be connected to the rear portion 1812 of the tractor 1810. The semi-trailer 1820 coupled to the tractor 1810 can be towed by the tractor 1810. To facilitate driving on curved roads, the semi-trailer 1820 can be rotatably coupled to the tractor 1810. For example, the tractor 1810 and the semi-trailer 1820 can be rotatably coupled using a coupling device including a steering wheel hook 1860 and a king pin 1880. However, the coupling mechanism between the tractor 1810 and the semi-trailer 1820 is not limited thereto.

[0326] The following will refer to Figure 19A 、 Figure 19B 、 Figure 19C and Figure 19D A wireless resource allocation method for direct communication between electronic devices required for executing distributed computing tasks among an electronic device 720 , a first other electronic device 722 , a second other electronic device 724 , and a third other electronic device 726 according to an embodiment of the present invention is described.

[0327] The direct communication according to an embodiment of the present invention may include communication between simple devices having a communication function and various forms of devices having a communication function, such as a smartphone or a personal computer.

[0328] Figure 19A 、 Figure 19B 、 Figure 19C and Figure 19D An example of various scenarios of wireless resource allocation for direct communication between electronic devices according to an embodiment is shown.

[0329] According to an embodiment of the present invention, the scenarios of direct communication between electronic devices applicable thereto can be roughly divided into (1) out-of-coverage network scenario; (2) partial-coverage network scenario; and (3) in-coverage network scenario according to whether the electronic device and the other electronic device are located within / outside the cell coverage.

[0330] Figure 19A An example of an out-of-coverage network scenario of direct communication according to an embodiment is shown. In the out-of-coverage network scenario, the electronic device 720 and other electronic devices 722, 724, and 726 can directly communicate with each other without the control of the eNodeB.

[0331] Figure 19B An example of a partial coverage network scenario of direct communication according to an embodiment is shown. In the partial coverage network scenario, direct communication is performed between an electronic device 720 located within the coverage of an eNodeB network and other electronic devices 722, 724, 726 located outside the coverage of the eNodeB network.

[0332] Figure 19C and Figure 19D An example of a network scenario within the coverage of direct communication according to an embodiment is shown. The network scenario within the coverage of direct communication according to an embodiment can be divided into a single-cell network scenario (In-Coverage-Single-Cell Network) according to whether the electronic device 720 to perform direct communication and other electronic devices 722, 724, 726 are located in the coverage of the same cell controlled by the same eNodeB. Figure 19C ) and In-Coverage-Multi-Cell Network scenarios ( Figure 19D ).

[0333] exist Figure 19C In the embodiment, the electronic device 720 and other electronic devices 722, 724, and 726 can perform direct communication under the control of the eNodeB within the same network coverage controlled by the same eNodeB.

[0334] exist Figure 19D In FIG, the electronic device 720 is located in a first cell coverage area (Cell coverage 1) managed by a first eNodeB, and the other electronic devices 722, 724, and 726 are located in a second cell coverage area (Cell coverage 2) managed by a second eNodeB that is different from the first eNodeB. Figure 19D FIG shows a scenario where the cell where the electronic device 720 is located and the cells where the other electronic devices 722, 724, and 726 are located are controlled by different eNodeBs. Figure 19D In the embodiment, the electronic device 720 and other electronic devices 722, 724, and 726 perform direct communication under the control of each eNodeB that manages the coverage of each cell (network coverage).

[0335] Figure 19A 、 Figure 19B 、 Figure 19C and Figure 19D An example of a direct communication scenario between electronic devices 720, 722, 724, and 726 according to an embodiment of the present invention is shown. In an actual direct communication environment between electronic devices 720, 722, 724, and 726, Figure 19A 、 Figure 19B 、 Figure 19C and Figure 19D The scenario may also be compounded.

[0336] Direct communication between electronic devices can generally be performed both within and outside the network coverage area. Links used for direct communication between electronic devices are generally referred to as direct links.

[0337] In one embodiment of the present invention, direct link transmission operates within the uplink spectrum in frequency division duplexing (FDD) and within uplink (or downlink) subframes in time division duplexing (TDD). Time division multiplexing (TDM) can be used to multiplex sidelink and uplink transmissions.

[0338] In one embodiment of the present invention, it is assumed that the sidelink transmission and the uplink transmission do not occur simultaneously. In addition, it is assumed that the sidelink transmission and the downlink transmission do not occur simultaneously.

[0339] The direct communication process can be roughly divided into a discovery process, a direct communication process, and a synchronization process.

[0340] 1) Discovery

[0341] exist Figure 7C In the example, electronic device 720 needs to perform the role of sending a discovery message. Therefore, electronic device 720 sends a discovery message, and other electronic devices 722, 724, and 726 receive the discovery message. The sending and receiving roles of electronic device 720 and other electronic devices 722, 724, and 726 can be interchanged. The transmission from electronic device 720 can be received by more than one electronic device, such as other electronic devices 722, 724, and 726.

[0342] The discovery message may include a single Media Access Control Protocol Data Unit (MAC PDU), wherein the single MAC PDU may include an electronic device ID (e.g., UE ID) and an application ID. The channel for transmitting the discovery message may be defined as a Physical Sidelink Discovery Channel (PSDCH). The structure of the PSDCH channel may reuse the structure of the Physical Uplink Shared Channel (PUSCH).

[0343] Furthermore, in one embodiment, the resource allocation method for direct communication discovery may utilize both the first type and the second type.

[0344] In the case of the first type (Type 1), the eNodeB may allocate resources for transmitting discovery messages in a non-electronic device specific (non-UE specific) manner. Specifically, according to one embodiment, a radio resource pool (Radio Resource Pool) consisting of multiple subframes for discovery transmission and reception may be allocated according to a specific period, and the electronic device 720 for sending the discovery message may arbitrarily select a specific resource in this radio resource pool and then send the discovery message. This periodic discovery resource pool may be allocated in a semi-static manner for transmitting discovery messages. The setting information of the discovery resource pool (Discovery Resource Pool) for discovery messages may include a discovery period, the number of subframes that can be used to transmit discovery messages within the discovery period (i.e., the number of subframes constituting the radio resource pool).

[0345] like Figure 19B 、 Figure 19C and Figure 19D As shown, in the case of an electronic device 720 located within the coverage of a cell, the discovery resource pool used to send the discovery message is set by the eNodeB, and the eNodeB can use Radio Resource Control (RRC) signaling (for example, System Information Block (SIB)) to notify the electronic device 720 of information about the discovery resource pool.

[0346] Within a discovery cycle, the discovery resource pool allocated for sending and receiving discovery messages can be multiplexed by time-frequency resource blocks of the same size through time division multiplexing (TDM) and / or frequency division multiplexing (FDM). These time-frequency resource blocks of the same size can be referred to as "discovery resources." The discovery resources can be used by an electronic device to transmit a discovery MAC PDU. The transmission of the MAC PDU transmitted by an electronic device can be repeated continuously (contiguous) or non-contiguously (noncontiguously) within the discovery cycle (i.e., the wireless resource pool).

[0347] In the second type, the discovery resources used to transmit discovery messages can be allocated specifically to the electronic device. The second type can be further divided into Type 2A (Type 2A) and Type 2B (Type 2B). Type 2A allocates resources each time the electronic device transmits a discovery message during the discovery cycle, while Type 2B allocates resources semi-persistently.

[0348] In the case of type 2B, the electronic device that establishes an RRC connection (RRC_CONNECTED) with the eNodeB requests the eNodeB to allocate resources for transmitting discovery messages through RRC signaling. In addition, the eNodeB can allocate resources through RRC signaling. When the electronic device switches to the Radio Resource Control Idle (RRC_IDLE) state or the eNodeB withdraws the resource allocation through RRC signaling, the electronic device can release the most recently allocated resources. As described above, in the case of type 2B, wireless resources are allocated through RRC signaling, and the activation / deactivation of the allocated wireless resources is determined by PDCCH. The wireless resource pool for receiving discovery messages can be set by the eNodeB, and the eNodeB can use RRC signaling (e.g., SIB) to notify the electronic device of the wireless resource pool for receiving discovery messages.

[0349] The electronic devices 722 , 724 , and 726 for receiving discovery messages may monitor the first type and second type discovery resource pools to receive discovery messages.

[0350] 2) Direct communication

[0351] When electronic device 720 has the role of transmitting data via direct communication, electronic device 720 directly transmits the data, and first other electronic device 722, second other electronic device 724, and third other electronic device 726 directly receive the data from electronic device 720 via direct links. The transmitting and receiving roles of electronic device 720 and first other electronic device 722, second other electronic device 724, and third other electronic device 726 can be interchanged. Direct data transmission from electronic device 720 can be received by more than one electronic device, such as first other electronic device 722, second other electronic device 724, and third other electronic device 726.

[0352] The sending and receiving of discovery messages and the sending and receiving of data for direct communication can be defined independently without being associated with each other. That is, in direct communication in multicast and broadcast modes, the sending and receiving process of discovery messages may not be required. As described above, when the discovery process (sending and receiving of discovery messages, sending and receiving of response messages thereof) and the sending and receiving process of data performed by direct communication are defined independently, electronic devices 720, 722, 724, 726 do not need to identify neighboring electronic devices. In other words, in the case of multicast and broadcast direct communication, all receiving electronic devices in the group (electronic devices that receive messages) are not required to be close to each other.

[0353] According to one embodiment, a channel used for transmitting data via direct communication between the electronic devices 720, 722, 724, and 726 may be defined as a Physical Sidelink Shared Channel (PSSCH). Furthermore, a channel used for transmitting control information for direct communication (e.g., scheduling assignments (SA) for transmitting direct communication data, transmission format, etc.) may be defined as a Physical Sidelink Control Channel (PSCCH).

[0354] Furthermore, according to an embodiment, a resource allocation method for direct communication between electronic devices 720 , 722 , 724 , and 726 may utilize a first mode (mode 1) and a second mode (mode 2).

[0355] The first mode refers to the way in which the eNodeB schedules the resources used by the electronic device to transmit data or control information for direct communication. Figure 19C and Figure 19DAs shown, electronic devices 720, 722, 724, and 726 within the coverage area can apply the first mode. In the first mode, the eNodeB sets the resource pool required for direct communication. The resource pool required for direct communication can be divided into a control information pool and a data pool. The eNodeB uses the data control channel to schedule control information and data transmission resources in the set pool to the sending electronic device (the electronic device to send data). The sending electronic device uses the allocated resources to send control information and data to other receiving electronic devices.

[0356] In the first mode, the transmitting electronic device requests transmission resources from the eNodeB, which can schedule resources for transmitting control information and direct communication data. Specifically, in the first mode, the transmitting electronic device must be in the Radio Resource Control (RRC CONNECTED) state to perform direct communication. The transmitting electronic device transmits a scheduling request to the eNodeB, and in response, a Buffer Status Report (BSR) procedure may be performed to allow the eNodeB to determine the amount of resources requested by the transmitting electronic device.

[0357] In addition, in the first mode, the receiving electronic device monitors the control information pool and, after decoding the control information related to itself, can selectively decode the data transmission related to the control information. The receiving electronic device can also not decode the data pool based on the control information decoding result.

[0358] Second mode: The electronic device arbitrarily selects specific resources from the resource pool to transmit data or control information for direct communication. The second mode can be applied to Figure 19A and Figure 19B Other electronic devices 722, 724, 726 in.

[0359] In the second mode, the resource pool used for transmitting control information and / or the resource pool used for data transmission via direct communication can be pre-set or semi-statically set. The electronic device receives the set resource pool (time and frequency) and can select resources from the resource pool for direct communication transmission. In other words, the electronic device can select resources for control information transmission from the control information resource pool to transmit the control information. In addition, the electronic device can select resources from the data resource pool for direct communication data transmission.

[0360] 3) Synchronization

[0361] According to one embodiment, a synchronization signal (or sidelink synchronization signal) for direct communication between electronic devices 720, 722, 724, and 726 may be used by the electronic devices 720, 722, 724, and 726 to achieve time-frequency synchronization. In particular, when an electronic device is outside of network coverage, it is not under the control of the eNodeB. Therefore, new signals and procedures for establishing synchronization between electronic devices may be defined.

[0362] An electronic device that periodically transmits a synchronization signal for performing direct communication between electronic devices 720, 722, 724, and 726 may be referred to as a direct communication synchronization source. When the direct communication synchronization source is an eNodeB, the structure of the transmitted direct communication synchronization signal may be the same as the primary synchronization signal (PSS) / secondary synchronization signal (SSS). When the direct communication synchronization source is not an eNodeB (for example, an electronic device or a satellite, etc.), the structure of the transmitted direct communication synchronization signal may be redefined. The direct communication synchronization signal may be periodically transmitted with a period of not less than 40ms. Each electronic device may have multiple physical layer sidelink synchronization identities.

[0363] Before transmitting a direct communication synchronization signal, the electronic device may first search for a direct communication synchronization source. Furthermore, upon finding a direct communication synchronization source, the electronic device may acquire time and frequency synchronization by receiving the direct communication synchronization signal from the searched direct communication synchronization source. Furthermore, the electronic device that has acquired time and frequency synchronization may transmit a direct communication synchronization signal to other electronic devices.

[0364] For the sake of convenience, direct communication between two electronic devices will be used as an example of direct communication between electronic devices below, but the scope of the present invention is not limited to this. The same principle described in one embodiment can also be applied to direct communication between more than two electronic devices.

[0365] According to one embodiment, all electronic devices may perform a discovery process in a decentralized manner (hereinafter referred to as a "decentralized discovery process"). The decentralized manner of performing a direct communication discovery process means that, unlike a centralized manner, resource selection is not determined by one node (e.g., eNB, electronic device, etc.), but all electronic devices independently determine and select discovery resources in a decentralized manner, and send and receive discovery messages through the selected discovery resources. According to one embodiment, in a decentralized discovery manner, a discovery subframe (i.e., a "discovery resource pool") for the discovery process may be fixed (or dedicated) in the entire cellular uplink frequency-time resource, and the remaining area may constitute an existing LTE uplink wide area network (WAN) subframe area.

[0366] The discovery resource pool may be composed of more than one subframe. In addition, the discovery resource pool may be periodically allocated at a certain time interval (ie, a "discovery period"). Furthermore, the discovery resource pool may be repeatedly configured within a discovery period.

[0367] According to an embodiment, the electronic device 720 may independently select resources (ie, “discovery resources”) for transmitting its own discovery message within the dedicated allocated discovery pool, and transmit the discovery message through the selected resources.

[0368] According to an embodiment, the discovery process of the electronic device 720 may be composed of a three-step process including a resource sensing step for discovery message transmission, a resource selection step for discovery message transmission, and a discovery message sending and receiving step.

[0369] First, in the resource sensing step, all electronic devices performing the discovery process receive (i.e., sense) all discovery messages in a decentralized manner (i.e., independently) within one period (1 period) of the discovery resource (i.e., the discovery resource pool). In addition, in the resource selection step, the electronic device classifies low-energy resources from the sensed resources and randomly selects discovery resources within a certain range. At this time, the discovery resource can be composed of more than one resource block of the same size and can be multiplexed within the discovery resource pool through TDM and / or FDM.

[0370] According to one embodiment, in the resource selection step, the reason why the electronic device selects a resource with a low energy level as a discovery resource is because, in the case of a resource with a low energy level, it can be explained that the surrounding electronic devices do not use a large number of the same discovery resources for direct communication. That is, this indicates that there are not too many terminals performing a direct communication discovery process that interferes with the surrounding environment. Therefore, when these low-energy-level resources are selected, the probability of less interference when sending a discovery message is higher. In addition, in one embodiment, the reason why the electronic device does not select the resource with the lowest energy level, but randomly selects the discovery resource within a predetermined range is because, if the resource with the lowest energy level is selected, multiple terminals may simultaneously select the same resource with the lowest energy level. That is, selecting the same resource with the lowest energy level may cause a lot of interference. Therefore, preferably, random selection is performed within a predetermined range (i.e., within the candidate pool constituting the optional resources).

[0371] In addition, in the discovery message transmission and reception, the electronic device transmits and receives the discovery message based on the selected discovery resource after one discovery cycle, and may periodically transmit and receive the discovery message according to a random resource hopping mode in a subsequent discovery cycle.

[0372] As described above, the direct communication discovery process according to one embodiment can be performed while the electronic device is connected to the eNodeB, i.e., in the RRC connected state, or can be continued while the electronic device is not connected to the eNodeB, i.e., in the RRC idle state. Considering the above discovery method, all electronic devices can sense all resources being transmitted by other electronic devices in the surrounding area (i.e., the discovery resource pool) and randomly select discovery resources from them within a certain range.

[0373] The electronic device adaptively determines the size of the discovery resource region according to the energy level, but may also arbitrarily determine the location of the discovery resource region. For example, the electronic device may arbitrarily determine the location of the discovery resource region using its own identifier.

[0374] In addition, the starting position of the discovery resource region in the frequency domain, time domain, or spatial domain can be fixed. That is, the size of the discovery resource region can be variably determined based on the detected energy level, but can start from a preset point in the frequency domain, time domain, or spatial domain. For example, when the discovery resource region is adaptively determined in the frequency domain, the location of the discovery resource region can be determined starting from a pre-set physical resource block (PRB) index in the discovery resource pool.

[0375] In addition, according to one embodiment, the electronic device can sense the discovery resources within the determined discovery resource area and select resources for transmitting discovery messages from the sensed discovery resources. That is, the electronic device can receive (i.e., sense) all discovery messages transmitted in the selected discovery resources, and classify resources with low energy levels, and randomly select discovery resources within a certain range thereof. In addition, the electronic device can send a discovery message through the selected discovery resource, and periodically send and receive discovery messages according to a random resource hopping mode in a subsequent discovery cycle.

[0376] According to one embodiment, the discovery resource region may be dynamically configured for each discovery resource pool and semi-statically configured for one or more discovery cycles. Furthermore, the discovery resource region may be configured cell-specifically and applied commonly to electronic devices belonging to the cell, or may be configured UE-specifically for each electronic device.

[0377] The configuration information for the discovery resource region configured as described above may be periodically broadcast to the electronic device as system information such as a system information block (SIB) and a master information block (MIB). Alternatively, the configuration information may be transmitted to the electronic device via RRC signaling or a physical layer channel (e.g., PDCCH or PDSCH).

[0378] An electronic device configured for a leading vehicle in a convoy of vehicles is provided. The electronic device may include: a processor, a memory for storing instructions, and a camera. When executed by the processor, the instructions may cause the electronic device to perform the following actions: using the camera to acquire a front image in front of the leading vehicle, generating a cropped image by cropping the front image, sending at least one of the front image and the cropped image to each other electronic device configured in each following vehicle, acquiring first information related to an external object from the external object included in at least one of the front image and the cropped image, receiving second information related to the external object acquired from the external object included in at least the other of the front image and the cropped image from the other electronic device, and identifying the external object based on the first information and the second information.

[0379] According to one embodiment, the front image may include a first image based on a first perspective, and the cropped image may include: a second image based on a second perspective generated by cropping a first area of ​​the first image, and a third image based on a third perspective generated by cropping the second area of ​​the first image.

[0380] According to one embodiment, when executed by the processor, the instructions may cause the electronic device to perform the following actions: transmit at least one of the second image and the third image to each of the other electronic devices. The first information may include third information related to the external object included in the first image. The second information may include at least one of fourth information related to the external object identified from the external object included in the second image and fifth information related to the external object identified from the external object included in the third image, from the other electronic device.

[0381] According to one embodiment, when the instructions are executed by the processor, the electronic device may perform the following actions: sending the second image to a first other electronic device in a first following vehicle configured among the following vehicles; sending the third image to a second other electronic device in a second following vehicle configured among the following vehicles; receiving the fourth information from the first other electronic device; receiving the fifth information from the second other electronic device; and identifying the external object based on the third information, the fourth information, and the fifth information.

[0382] According to an embodiment, the second viewing angle may be wider than the third viewing angle and narrower than the first viewing angle.

[0383] According to one embodiment, when the processor executes the instructions, the electronic device may perform the following actions: setting a region of interest based on the vanishing point of the front image; and cropping the front image based on the region of interest to generate the cropped image.

[0384] According to one embodiment, when the instruction is executed by the processor, it can cause the electronic device to perform the following actions: divide the front image and the cropped image into a first main image processed by the electronic device based on a first FPS and a first sub-image processed by the electronic device based on a second FPS lower than the first FPS; obtain the first information based on the first main image and the first sub-image.

[0385] According to one embodiment, when the instruction is executed by the processor, it can cause the electronic device to perform the following actions: divide the front image and the cropped image into a second main image processed by each of the other electronic devices based on the first FPS and a second sub-image processed by each of the other electronic devices based on the second FPS; send the front image and the cropped image to each of the other electronic devices; receive the second information obtained based on the second main image and the second sub-image from the other electronic devices.

[0386] According to an embodiment, the first main image may be an image different from the second main image.

[0387] According to one embodiment, when the instruction is executed by the processor, it can cause the electronic device to perform the following actions: when it is identified that the external object identified based on the first information is different from the external object identified based on the second information, identify the image including the external object; identify whether the image is the first main image or the second main image; when the image is identified as the first main image, identify the external object based on the first information; when the image is identified as the second main image, identify the external object based on the second information.

[0388] According to one embodiment, the first information and the second information may be acquired using an image processing model (eg, a neural network model).

[0389] An electronic device configured for a leading vehicle in a convoy of vehicles is provided. The electronic device may include: a processor, a memory for storing instructions, and a camera. When executed by the processor, the instructions may cause the electronic device to perform the following actions: using each of the cameras to acquire a front image in front of the leading vehicle, transmitting the front image to each other electronic device configured in each following vehicle, acquiring first information related to an external object included in at least one of the front images, receiving second information related to the external object acquired from the external object included in at least one of the front images from the other electronic devices, and identifying the external object based on the first and second information.

[0390] According to one embodiment, the camera may include: a first camera having a first viewing angle, a second camera having a second viewing angle, and a third camera having a third viewing angle. The front image may include: a first image based on the first viewing angle acquired by the first camera, a second image based on the second viewing angle acquired by the second camera, and a third image based on the third viewing angle acquired by the third camera.

[0391] According to one embodiment, when executed by the processor, the instructions may cause the electronic device to perform the following actions: transmit at least one of the second image and the third image to each of the other electronic devices. The first information may include third information related to the external object included in the first image. The second information may include at least one of fourth information related to the external object identified from the external object included in the second image and fifth information related to the external object identified from the external object included in the third image, from the other electronic device.

[0392] According to one embodiment, when the instructions are executed by the processor, the electronic device may perform the following actions: sending the second image to a first other electronic device in a first following vehicle configured among the following vehicles; sending the third image to a second other electronic device in a second following vehicle configured among the following vehicles; receiving the fourth information from the first other electronic device; receiving the fifth information from the second other electronic device; and identifying the external object based on the third information, the fourth information, and the fifth information.

[0393] According to an embodiment, when the processor executes the instructions, the electronic device may perform the following actions: change the second image and the third image to specified sizes and send the sizes.

[0394] According to one embodiment, when the instructions are executed by the processor, the electronic device may perform the following actions: transmit the first image, the second image, and the third image to a first other electronic device in a first of the following vehicles and a second other electronic device in a second of the following vehicles. The first information may include third information related to the external object obtained from the first image, the second image, and the third image. The second information may include fourth information related to the external object obtained from the first image, the second image, and the third image by the first other electronic device, and fifth information related to the external object obtained from the first image, the second image, and the third image by the second other electronic device.

[0395] According to one embodiment, when the instruction is executed by the processor, it can cause the electronic device to perform the following actions: divide the front image into a first main image processed by the electronic device based on a first FPS and a first sub-image processed by the electronic device based on a second FPS lower than the first FPS; obtain the third information based on the first main image and the first sub-image.

[0396] According to one embodiment, when the instruction is executed by the processor, it can cause the electronic device to perform the following actions: divide the front image into a second main image processed by each of the other electronic devices based on the first FPS and a second sub-image processed by each of the other electronic devices based on the second FPS; assign the second main image and the second sub-image to each of the other electronic devices; and receive the second information obtained based on the second main image and the second sub-image from the other electronic devices.

[0397] A method is provided for an electronic device configured in a leading vehicle in a convoy of vehicles. The method may include the following steps: using a camera of the electronic device to acquire a front image in front of the leading vehicle, cropping the front image to generate a cropped image, transmitting at least one of the front image and the cropped image to each other electronic device configured in each following vehicle, acquiring first information related to an external object from the external object included in at least one of the front image and the cropped image, receiving second information related to the external object acquired from the external object included in at least the other of the front image and the cropped image from the other electronic device, and identifying the external object based on the first and second information.

Claims

1. An electronic device configured for a leading vehicle in a convoy of vehicles, wherein: The electronic device comprises: processor, memory for storing instructions, and Camera; When executed by the processor, the instructions cause the electronic device to perform the following actions: Using the camera to obtain the front image in front of the leading vehicle, Generate a cropped image by cropping the front image, sending at least one of the front image and the cropped image to each other electronic device configured in each following vehicle, acquiring first information related to the external object from the external object included in at least one of the front image and the cropped image, receiving, from the other electronic device, second information related to the external object obtained from the external object included in at least the other of the front image and the cropped image, The external object is identified based on the first information and the second information.

2. The electronic device according to claim 1, wherein The front image includes a first image based on a first perspective; The cropped image includes: a second image based on a second perspective generated by cropping a first area of ​​the first image, and A third image based on a third perspective is generated by cropping the second area of ​​the first image.

3. The electronic device according to claim 2, wherein: When executed by the processor, the instructions cause the electronic device to perform the following actions: sending at least one of the second image and the third image to each of the other electronic devices; The first information includes third information related to the external object from the external object included in the first image, The second information includes at least one of fourth information related to the external object recognized from the external object included in the second image and fifth information related to the external object recognized from the external object included in the third image from the other electronic device.

4. The electronic device according to claim 3, wherein: When executed by the processor, the instructions cause the electronic device to perform the following actions: sending the second image to a first other electronic device configured in a first following vehicle among the following vehicles; sending the third image to a second other electronic device in a second following vehicle configured in the following vehicle; receiving the fourth information from the first other electronic device; receiving the fifth information from the second other electronic device; The external object is recognized based on the third information, the fourth information, and the fifth information.

5. The electronic device according to claim 2, wherein: The second viewing angle is wider than the third viewing angle and narrower than the first viewing angle. The electronic device according to claim 1 , wherein: When executed by the processor, the instructions cause the electronic device to perform the following actions: Setting a region of interest based on the vanishing point of the front image; The front image is cropped based on the region of interest to generate the cropped image.

7. The electronic device according to claim 1, wherein When executed by the processor, the instructions cause the electronic device to perform the following actions: dividing the front image and the cropped image into a first main image processed by the electronic device based on a first frame rate per second and a first sub-image processed by the electronic device based on a second frame rate per second lower than the first frame rate per second; The first information is acquired based on the first main image and the first sub-image.

8. The electronic device according to claim 7, wherein: When executed by the processor, the instructions cause the electronic device to perform the following actions: dividing the front image and the cropped image into a second main image processed by each of the other electronic devices based on the first frame rate per second and a second sub-image processed by each of the other electronic devices based on the second frame rate per second; sending the front image and the cropped image to each of the other electronic devices; The second information obtained based on the second main image and the second sub image is received from the other electronic device.

9. The electronic device according to claim 8, wherein: The first main image is an image different from the second main image.

10. The electronic device according to claim 8, wherein When executed by the processor, the instructions cause the electronic device to perform the following actions: When it is recognized that the external object identified based on the first information is different from the external object identified based on the second information, identifying an image including the external object; identifying whether the image is the first main image or the second main image; When the image is identified as the first main image, identifying the external object based on the first information; When the image is identified as the second main image, the external object is identified based on the second information.

11. The electronic device according to claim 1, wherein: The memory includes: An image processing model is used to extract the external object from the front image and the cropped image.

12. An electronic device configured for a leading vehicle in a convoy of vehicles, wherein: The electronic device comprises: processor, memory for storing instructions, and Camera; When executed by the processor, the instructions cause the electronic device to perform the following actions: Utilizing each of the cameras to obtain a front image in front of the leading vehicle, sending the forward image to each other electronic device configured in each following vehicle, acquiring first information related to the external object from at least one external object included in the front image, receiving, from the other electronic device, second information related to the external object obtained from the external object included in at least one of the front images, The external object is identified based on the first information and the second information.

13. The electronic device according to claim 12, wherein: The camera includes: a first camera having a first viewing angle, a second camera having a second viewing angle, and a third camera with a third perspective; The front image includes: a first image based on the first perspective acquired by the first camera, a second image based on the second perspective acquired by the second camera, and A third image based on the third perspective acquired by the third camera.

14. The electronic device according to claim 13, wherein: When executed by the processor, the instructions cause the electronic device to perform the following actions: sending at least one of the second image and the third image to each of the other electronic devices; The first information includes third information related to the external object from the external object included in the first image, The second information includes at least one of fourth information related to the external object recognized from the external object included in the second image and fifth information related to the external object recognized from the external object included in the third image from the other electronic device.

15. The electronic device according to claim 14, wherein: When executed by the processor, the instructions cause the electronic device to perform the following actions: sending the second image to a first other electronic device configured in a first following vehicle among the following vehicles; sending the third image to a second other electronic device in a second following vehicle configured in the following vehicle; receiving the fourth information from the first other electronic device; receiving the fifth information from the second other electronic device; The external object is recognized based on the third information, the fourth information, and the fifth information.

16. The electronic device according to claim 14, wherein: When executed by the processor, the instructions cause the electronic device to perform the following actions: The second image and the third image are changed to a specified size and transmitted.

17. The electronic device according to claim 13, wherein: When executed by the processor, the instructions cause the electronic device to perform the following actions: transmitting the first image, the second image, and the third image to a first other electronic device disposed in a first following vehicle among the following vehicles and a second other electronic device disposed in a second following vehicle among the following vehicles; The first information includes third information related to the external object obtained from the first image, the second image, and the third image. The second information includes: fourth information related to the external object obtained from the first other electronic device from the first image, the second image, and the third image, and Fifth information related to the external object obtained from the first image, the second image, and the third image by the second other electronic device.

18. The electronic device according to claim 13, wherein: When executed by the processor, the instructions cause the electronic device to perform the following actions: dividing the front image into a first main image processed by the electronic device based on a first frame rate per second and a first sub-image processed by the electronic device based on a second frame rate per second lower than the first frame rate per second; The third information is acquired based on the first main image and the first sub-image.

19. The electronic device according to claim 18, wherein: When executed by the processor, the instructions cause the electronic device to perform the following actions: dividing the front image into a second main image processed by each of the other electronic devices based on the first frame rate per second and a second sub-image processed by each of the other electronic devices based on the second frame rate per second; assigning the second main image and the second sub-image to each of the other electronic devices; The second information obtained based on the second main image and the second sub image is received from the other electronic device.

20. A method of configuring an electronic device of a lead vehicle in a convoy of vehicles, wherein: The steps include: Using the camera of the electronic device to obtain the front image in front of the leading vehicle, Generate a cropped image by cropping the front image, sending at least one of the front image and the cropped image to each other electronic device configured in each following vehicle, acquiring first information related to the external object from the external object included in at least one of the front image and the cropped image, receiving, from the other electronic device, second information related to the external object acquired from the external object included in at least the other of the front image and the cropped image, and The external object is identified based on the first information and the second information.

Images