Method of outputting analog signal and electronic device for performing the same

An electronic device processes vehicle camera signals to generate composite video and detect hazards, enhancing driver safety by integrating camera data with vehicle operation information without replacing existing systems.

JP2024159384AActive Publication Date: 2024-11-08NC& CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023115961
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-04-28
Filing Date
2023-07-14
Publication Date
2024-11-08
Estimated Expiration
2043-07-14

AI Technical Summary

Technical Problem

There is a need for an electronic device that can assist drivers by enhancing their field of vision and detecting dangerous situations using vehicle-mounted cameras without replacing existing photography and monitoring systems, and transmit these images to a device within the vehicle.

Method used

An electronic device installed in a vehicle processes analog signals from vehicle cameras, converts them into digital signals, applies video recognition models to identify target objects, generates a composite video with computer graphics, and outputs an analog signal to a receiving device, which can display or store the composite image, while also determining potential hazards based on vehicle operation information.

Benefits of technology

The device enhances driver awareness of potential hazards by providing real-time composite video and alerts, improving safety by integrating camera data with vehicle operation information without replacing existing systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159384000001_ABST
    Figure 2024159384000001_ABST
Patent Text Reader

Abstract

To provide an electronic device and a method of outputting an analog signal using the electronic device installed in a vehicle.SOLUTION: Provided is an electronic device configured to receive an analog signal for an original image captured using a target camera mounted on a vehicle, convert the analog signal for the original image into a digital signal based on a resolution of the original image, generate a digital image based on the digital signal, determine a target object in the digital image using one or more image recognition models, generate a synthesized image by synthesizing a computer graphic for the target object with the digital image, convert the synthesized image into an analog signal, and transmit the analog signal for the synthesized image to an analog signal-receiving device installed in the vehicle.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a method and system for outputting an analog signal of a composite image using an electronic device. [Background technology]

[0002] For safe vehicle operation, a function is provided that outputs and stores surrounding images captured by a camera installed in the vehicle via a device inside the vehicle. There is a need for a means that not only helps the driver to maintain a clear field of vision while driving, but also detects and determines dangerous situations in the vehicle and displays the detected situations in the captured images to notify the driver. There is a need for a means that transmits images notifying dangerous situations to a device inside the vehicle by installing an electronic device in a data transmission path without replacing the existing imaging and monitoring means installed in the vehicle. Summary of the Invention [Problem to be solved by the invention]

[0003] An electronic device according to an embodiment provides an analog signal output method using an electronic device provided in a vehicle. [Means for solving the problem]

[0004] In one embodiment, a method for outputting an analog signal of a composite image performed by an electronic device installed in a vehicle includes operations of receiving an analog signal for an original image captured using a target camera mounted on the vehicle, converting the analog signal for the original image into a digital signal based on a resolution of the original image, generating a digital image based on the digital signal, determining a target object in the digital image using one or more image recognition models, generating a composite image by combining computer graphics for the target object with the digital image, converting the composite image into an analog signal, and transmitting the analog signal for the composite image to an analog signal receiving device installed in the vehicle.

[0005] The operation of generating a digital image based on the digital signal may include an operation of generating the digital image having a format of raw data or YUV data based on the digital signal.

[0006] The operation of determining a target object in the digital image using the one or more image recognition models may include an operation of generating first processing data for a deep learning image recognition method and second processing data for a computer vision image recognition method based on the digital image, an operation of generating deep learning recognition data for the first processing data using a deep learning image recognition model, an operation of generating computer vision recognition data for the second processing data using a computer vision image recognition model, and an operation of determining a target object in the digital image based on the deep learning recognition data and the computer vision recognition data.

[0007] The operation of generating the first processed data and the second processed data, respectively, based on the digital image may include an operation of generating the first processed data and the second processed data, respectively, by applying at least one process of color format conversion, filtering, noise reduction, cropping, or scaling to the digital image.

[0008] The operation of generating the deep learning recognition data for the first processed data using the deep learning image recognition model may include an operation of determining target object information for the first processed data using the deep learning image recognition model, and an operation of generating the deep learning recognition data by determining whether the target object is detected in the digital image corresponding to a current frame based on previous object information for a previous frame within a predetermined frame range and the target object information.

[0009] The target object information may include at least one of a type, coordinates, shape, or a score indicating a recognition accuracy of the target object.

[0010] The operation of generating the deep learning recognition data by determining whether the target object is detected in the digital image may include an operation of determining whether the target object is detected in the digital image if the score for the target object in the current frame is less than a predetermined first threshold and an average value of previous scores for the target object in the previous frames is greater than or equal to a predetermined second threshold.

[0011] The operation of generating computer vision recognition data for the second processed data using the computer vision image recognition model may include an operation of generating the computer vision recognition data for the second processed data using the computer vision image recognition model, the computer vision recognition data including correction information based on camera parameters for the camera.

[0012] The operation of generating a composite image by synthesizing a computer graphic for the target object with the digital image may include an operation of generating a marker image layer as the computer graphic, the marker image layer displaying the position of the target object based on target object information included in the deep learning recognition data and correction information included in the computer recognition data, and an operation of generating the composite image by synthesizing the digital image and the computer graphic.

[0013] The analog signal receiving device may be a black box module.

[0014] The method for outputting an analog signal of a synthetic image may further include an operation of receiving operation information of the vehicle, and an operation of determining whether or not there is a risk factor for the current state of the vehicle based on the vehicle operation information and the deep learning recognition data.

[0015] The operation of receiving the vehicle operation information may include at least one of an operation of receiving operation information of the vehicle from an operation information relay module, or an operation of receiving position information of the vehicle from a GPS module.

[0016] The method for outputting an analog signal of a composite image may further include outputting a danger alarm when the danger element exists.

[0017] According to one embodiment, an electronic device for outputting an analog signal of a composite image includes a first conversion unit that receives an analog signal for an original image captured using a target camera mounted on a vehicle; a signal processing unit that converts the analog signal for the original image into a digital signal based on a resolution of the original image; an image processing unit that generates a digital image based on the digital signal; an image recognition unit that determines a target object in the digital image using one or more image recognition models; and a hazard detection software unit that generates a composite image by synthesizing a computer graphic of the target object with the digital image, converts the composite image into an analog signal, and transmits the analog signal for the composite image to an analog signal receiving device installed in the vehicle.

[0018] The image recognition unit may perform the following operations: generating first processing data for a deep learning image recognition method and second processing data for a computer vision image recognition method based on the digital image; generating deep learning recognition data for the first processing data using a deep learning image recognition model; generating computer vision recognition data for the second processing data using a computer vision image recognition model; and determining a target object in the digital image based on the deep learning recognition data and the computer vision recognition data.

[0019] The operation of generating the deep learning recognition data for the first processed data using the deep learning image recognition model may include an operation of determining target object information for the first processed data using the deep learning image recognition model, and an operation of generating the deep learning recognition data by determining whether the target object is detected in the digital image corresponding to a current frame based on previous object information for a previous frame within a predetermined frame range and the target object information.

[0020] The target object information may include at least one of a type, coordinates, shape, or a score indicating a recognition accuracy of the target object.

[0021] The operation of generating the deep learning recognition data by determining whether the target object is detected in the digital image may include an operation of determining whether the target object is detected in the digital image if the score for the target object in the current frame is less than a predetermined first threshold and an average value of previous scores for the target object in the previous frames is greater than or equal to a predetermined second threshold.

[0022] The operation of generating a composite image by synthesizing a computer graphic for the target object with the digital image may include an operation of generating a marker image layer displaying the position of the target object as the computer graphic based on target object information included in the deep learning recognition data and correction information included in the computer recognition data, and an operation of generating the composite image by synthesizing the digital image and the computer graphic. Effect of the Invention

[0023] According to the present invention, an electronic device can provide an analog signal output method using an electronic device provided in a vehicle. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 illustrates an analog signal output system according to an embodiment. [Diagram 2] FIG. 1 is a block diagram of an electronic device according to an embodiment. [Diagram 3] 1 is a flowchart of a method for outputting an analog signal of a composite image according to an embodiment. [Figure 4] 1 is a flowchart of a target object determination method according to an embodiment. [Diagram 5] 1 is a flowchart of a deep learning recognition data generation method according to an embodiment. [Figure 6] 1 is a flowchart of a synthetic image generating method according to an embodiment. [Figure 7] FIG. 13 is a diagram for explaining a synthetic image according to an example. [Figure 8] 10 is a flowchart illustrating reception of operation information according to an embodiment. [Figure 9] 4 is a flowchart of a danger warning output method according to an embodiment. [Figure 10] FIG. 1 is a block diagram of an electronic device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The specific structural or functional description of the embodiments is disclosed for the purpose of illustration only, and may be modified in various forms. Therefore, the embodiments are not limited to the specific disclosed forms, and the scope of the present specification includes modifications, equivalents, or alternatives within the technical spirit.

[0026] Although terms such as first or second may be used to describe multiple components, such terms should be construed only for the purpose of distinguishing one component from the other components. For example, a first component may be named a second component, and similarly, a second component may be named a first component.

[0027] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that although it is directly coupled or connected to the other component, there may be other components in between.

[0028] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, the terms "comprise" or "have" and the like indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0029] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as ideal or overly formal unless expressly defined in this specification.

[0030] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the drawings, the same components are given the same reference numerals regardless of the reference numerals, and redundant description thereof will be omitted.

[0031] FIG. 1 is a diagram showing an analog signal output system according to an embodiment.

[0032] The analog signal output system 100 according to one embodiment includes an electronic device 110, a camera 120, an analog signal receiving device 130, and an external device 140. The electronic device 110, the camera 120, the analog signal receiving device 130, and the external device 140 may be provided in a vehicle (not shown).

[0033] According to an embodiment, the camera 120 indicates a target camera connected to the electronic device 110 among one or more camera modules installed inside and / or outside the vehicle. The analog signal output system 100 includes an electronic device module corresponding to each camera module installed in the vehicle. For example, the camera 120 is a camera that captures the front or rear of the vehicle. For example, the camera 120 may be a camera that captures the left or right side of the vehicle.

[0034] According to an embodiment, the camera 120 outputs an analog signal for the captured original image, and the analog signal receiving device 130 can output the received analog signal in real time via a display or store it in a memory. For example, the analog signal receiving device 130 may represent a black box module installed in a vehicle. The electronic device 110 may be installed between the camera 120 and the analog signal receiving device 130. The electronic device 110 receives the analog signal output by the camera 120, generates a digital composite image based on the received analog signal, and transmits an analog signal for the digital composite image to the analog signal receiving device 130. For example, the camera 120 outputs an analog signal for the captured original image in a Composite Video Baseband Signal (CVBS), Turbo Video Interface (TVI), or Analog High Definition (AHD) format, and the resolution of the original image may be FHD (1920x1080), HD (1280x720), or CVBS (720x480i). The electronic device 110 can transmit an analog signal for a digital composite image having the same format and resolution as the analog signal output by the camera 120 to the analog signal receiving device 130 .

[0035] According to an embodiment, the electronic device 110 includes a first conversion unit 150, a signal processing unit 160, an image processing unit 170, an image recognition unit 180, and a danger detection software unit 190. The electronic device 110 will be described in detail below with reference to FIG.

[0036] According to an embodiment, the external device 140 includes a driving information relay module 142 and a GPS module 144. The analog signal output system 100 may or may not selectively include the driving information relay module 142 and / or the GPS module 144. The electronic device 110 receives vehicle driving information from the external device 140. The driving information includes vehicle operation information and location information. The vehicle operation information may include one or more of the vehicle's left / right turn signal lamp information, reverse gear information, brake information, or speed information.

[0037] According to one embodiment, the electronic device 110 can receive vehicle operation information from the traffic information relay module 142. The electronic device 110 may also receive vehicle location information from the GPS module 142 (either directly or via the traffic information relay module 142).

[0038] According to an embodiment, the operation information relay module 142 may obtain information on whether the turn signal lamp is on, whether the reverse gear is changed, or the operation state of the brake pedal based on an analog signal obtained through a power cable of the turn signal lamp, reverse lamp, or brake lamp of the vehicle. If the operation information relay module 142 is not connected to the GPS module 144, the operation information relay module 142 may obtain speed information through a power cable connected to a speed sensor that measures the speed of the vehicle by counting the number of vehicle revolutions of the vehicle. If the operation information relay module 142 is connected to the GPS module 144, the operation information relay module 142 may obtain speed information by calculating the travel distance and time interval of the vehicle using the position information received from the GPS module 144. For example, the operation information relay module 142 may determine the force applied to the brake pedal and the brake application period by combining information obtained through the power cable of an additional sensor such as a speed sensor or other pressure sensor and information obtained through the power cable of the brake lamp.

[0039] According to an embodiment, the driving information relay module 142 may receive various information related to danger detection of the vehicle from an OBD (On-Board Diagnostics) system of the vehicle via CAN (Controller Area Network) communication. The driving information relay module 142 may receive operation information including one or more of left / right turn signal lamp information, reverse gear information, brake information, or speed information of the vehicle, as well as vehicle mileage, fuel consumption, driving time, vehicle position, engine status, etc., and is not limited to the above description. As a result, the electronic device may receive vehicle driving information from the OBD system or power cable via the driving information relay module 142.

[0040] FIG. 2 is a block diagram of an electronic device according to one embodiment.

[0041] According to an embodiment, the electronic device 110 includes a first conversion unit 150 , a signal processing unit 160 , an image processing unit 170 , an image recognition unit 180 , and a danger detection software unit 190 .

[0042] The first conversion unit 150 receives an analog signal for an original image captured from one or more camera modules installed in a vehicle. For example, the first conversion unit 150 may receive an analog signal in a format and resolution designated in advance by a user. For example, the first conversion unit 150 may estimate the resolution of the received analog signal. The first conversion unit 150 converts the received analog signal into a digital signal based on the format or resolution of the analog signal. For example, the first conversion unit 150 may convert the analog signal into a digital signal conforming to the BT.656 (ITU-BT.656 or CCIR-656) standard, and transmit the converted digital signal to the signal processing unit 160.

[0043] The signal processor 160 is an image signal processor (ISP). The signal processor 160 may generate a digital image having a format of raw data or YUV data based on a digital signal. The signal processor 160 may buffer the generated digital image. The digital image may be transmitted to the image processor 170 and the synthesizer 192.

[0044] The image processor 170 may generate first processed data for a deep learning image recognition method and second processed data for a computer vision image recognition method based on the digital image. The image processor 170 may generate the first processed data and the second processed data by applying at least one of color format conversion (RGB or YUV), filtering, noise reduction, cropping, and scaling to the digital image.

[0045] The image recognition unit 180 includes a deep learning recognition unit 182 and a computer vision recognition unit 184. The deep learning recognition unit 182 includes a deep learning unit 182-1 and a data processing unit 182-2. The computer vision recognition unit 184 includes a recognition assistance unit 184-1 and a correction unit 184-2. The recognition assistance unit 184-1 may be included selectively.

[0046] According to an embodiment, the image recognition unit 180 may generate deep learning recognition data and computer vision recognition data for driver status monitoring (DSM) and blind spot detection (BSD) for a captured image.

[0047] According to an embodiment, the deep learning recognition unit 182 generates deep learning recognition data for the first processing data using a pre-stored and / or trained deep learning video recognition model. The deep learning unit 182-1 determines target object information for the first processing data using the deep learning video recognition model. The target object information may include at least one of the type, coordinates, shape, or score indicating the recognition accuracy of the target object. The target object indicates an object that causes a dangerous situation for the vehicle among various objects recognized by the deep learning video recognition model. The data processing unit 182-2 may generate deep learning recognition data by determining whether a target object is detected in the digital image.

[0048] According to an embodiment, the deep learning recognition unit 182 may use a deep learning recognition model that is specifically trained depending on the installation position of the target camera to which the electronic device 110 is connected. For example, when the electronic device 110 is connected to a target camera that captures the front of the vehicle, the electronic device 110 may include the deep learning recognition unit 182 that uses a deep learning recognition model specialized for front-view image recognition. For example, when the electronic device 110 is connected to a target camera that captures the rear of the vehicle, the electronic device 110 may include the deep learning recognition unit 182 that uses a deep learning recognition model specialized for rear-view image recognition.

[0049] According to an embodiment, the computer vision recognition unit 184 may generate computer vision recognition data for the second processing data using a computer vision image recognition model stored and / or trained in advance. The recognition assistant unit 184-1 may re-determine target object information for the second processing data using a computer vision image recognition model to complement the deep learning recognition data generated by the deep learning recognition unit 182. For example, the recognition assistant unit 184-1 may re-determine target object information to detect a target object that cannot be detected by the deep learning recognition unit 182. The correction unit 184-2 may generate computer vision recognition data including correction information based on camera parameters of a camera (e.g., the camera 120 in FIG. 1) for the second processing data (or for the second processing data in which the target object information has been re-determined). The camera parameters may include characteristics of a sensor and a lens constituting the camera, or coordinates and an installation angle at which the camera is mounted on the vehicle.

[0050] The danger detection software unit 190 collects and stores deep learning recognition data, computer vision recognition data, camera parameters, and various information related to the danger detection of the vehicle (e.g., the measured distance between the vehicle and the target object, the acceleration of the vehicle, the excel rate operation of the vehicle, the driving information of the vehicle, etc.), and performs comprehensive danger detection based on all the data. The danger detection software unit 190 includes a synthesis unit 192, a danger detection unit 194, and a second conversion unit 196.

[0051] The synthesis unit 192 may generate a marker image layer that displays the position of a target object as computer graphics based on the deep learning recognition data and the computer vision recognition data. The synthesis unit 192 may generate a synthetic image by synthesizing the digital image generated by the signal processing unit 160 and the generated computer graphics. The generation of the synthetic image will be described in detail with reference to FIG. 7.

[0052] The danger detection unit 194 determines whether or not a risk factor exists for the current state of the vehicle based on the vehicle operation information and the deep learning recognition data. In determining whether or not a risk factor exists, the computer vision recognition data including the re-determined target object information and / or the correction information may also be used.

[0053] The second converter 196 may convert the composite image generated by the combiner 192 into an analog signal. For example, the second converter 196 may convert the composite image into an analog signal in CVBS or TVI format, and the resolution of the corresponding image may be FHD, HD, or CVBS. The second converter 196 may convert the composite image into an analog signal having the same format and resolution as the analog signal output by the camera 120.

[0054] FIG. 3 is a flowchart of a method for outputting an analog signal of a composite image according to an embodiment.

[0055] According to one embodiment, the following operations 310-390 are performed by an electronic device (eg, electronic device 110 of FIGS. 1 and 2).

[0056] In operation 310, the electronic device receives an analog signal for an original image captured using a target camera mounted on a vehicle.

[0057] According to one embodiment, the electronic device may receive an analog signal in a format and resolution pre-specified by a user, for example, the electronic device may receive an analog signal in a CVBS, TVI, or AHD format for an original captured image, and the resolution of the image may be FHD, HD, or CVBS.

[0058] According to one embodiment, the electronic device may estimate the resolution of a received analog signal.

[0059] In operation 320, the electronic device converts the analog signal for the original video into a digital signal based on the resolution of the original video. The electronic device may convert the analog signal for the original video into a digital signal based on a pre-specified resolution or an estimated resolution. For example, the electronic device may convert the analog signal into a digital signal in the BT.656 (ITU-BT.656 or CCIR-656) standard.

[0060] In operation 330, the electronic device generates a digital image based on the digital signal. The electronic device may generate a digital image having a format of raw data or YUV data based on the digital signal. The electronic device may buffer the generated digital image.

[0061] In operation 340, the electronic device determines a target object in the digital image using one or more image recognition models. A method for determining a target object is described in detail with reference to FIG.

[0062] In operation 350, the electronic device generates a composite image by compositing a computer graphic for the target object with the digital image. For example, the electronic device may generate a marker image layer as a computer graphic that displays the location of the target object. The electronic device may generate the composite image by combining the digital image and the computer graphic. For example, the electronic device may generate a composite image in which a frame marker is displayed at the location of the target object.

[0063] The electronic device converts the composite video to an analog signal in act 360. The electronic device may convert the composite video to an analog signal having the same format and resolution as the analog signal received in act 310.

[0064] In operation 370, the electronic device transmits an analog signal for the composite image to an analog signal receiving device (e.g., analog signal receiving device 130 in FIG. 1) provided in the vehicle. The analog signal receiving device receives the analog signal for the composite image and can output the image for the received analog signal in real time via a display or store it in a memory.

[0065] In operation 380, the electronic device receives vehicle operation information, including vehicle motion information and vehicle location information, as described in more detail below with reference to FIG.

[0066] According to one embodiment, operation 390 occurs after operation 340 is performed.

[0067] In operation 390, the electronic device determines whether or not a risk factor exists for the current state of the vehicle based on the vehicle operation information and the deep learning recognition data. For example, the electronic device determines that a risk factor exists for the forward or backward state of the vehicle when there is an object moving straight in the driving direction of the vehicle or an object located on the driving path. For example, the electronic device may determine that a risk factor exists for the left or right turn state of the vehicle when an object is located on the driving path in the left or right turn direction of the vehicle. For example, the electronic device may determine that a risk factor exists when an object that is a collision risk exists within a certain distance based on the speed and / or acceleration of the vehicle. For example, the electronic device may determine whether or not a risk factor exists based on the real-time coordinates and type of the target object and the driving behavior of the vehicle.

[0068] According to one embodiment, operation 390 occurs after operation 420 shown in FIG. 4 below.

[0069] FIG. 4 is a flowchart of a target object determination method according to an embodiment.

[0070] According to one embodiment, the following operations 410-440 are performed after operation 330 described above with reference to Figure 3 is performed. Operations 410-440 may be performed by an electronic device (e.g., electronic device 110 of Figures 1 and 2).

[0071] In operation 410, the electronic device generates first processed data for a deep learning image recognition method and second processed data for a computer vision image recognition method based on the digital image.

[0072] According to an embodiment, the electronic device may generate the first processed data and the second processed data by applying at least one of color format conversion, filtering, noise reduction, cropping, or scaling to the digital image. The first processed data is generated in a standard or form suitable for processing by a deep learning recognition unit (e.g., deep learning recognition unit 182 in FIGS. 1 and 2) of the electronic device, and the second processed data is generated in a standard or form suitable for processing by a computer vision recognition unit (e.g., computer vision recognition unit 184 in FIGS. 1 and 2) of the electronic device.

[0073] In operation 420, the electronic device generates deep learning recognition data for the first processed data using the deep learning image recognition model. The deep learning recognition data includes whether a target object is detected in the digital image. A method for generating deep learning recognition data will be described in detail with reference to FIG. 5.

[0074] In operation 430, the electronic device generates computer vision recognition data for the second processed data using the computer vision image recognition model. The electronic device generates computer vision recognition data for the second processed data using the computer vision image recognition model, the computer vision recognition data including correction information based on camera parameters for a camera (e.g., camera 120 of FIG. 1). The camera parameters may include characteristics of a sensor and lens constituting the camera or coordinates and installation angle at which the camera is mounted on the vehicle.

[0075] In operation 440, the electronic device determines a target object in the digital image based on the deep learning recognition data and the computer vision recognition data. The electronic device may determine at least one of the type, coordinates, shape, or score indicating the recognition accuracy of the target object. The electronic device may determine a corrected target object based on the camera parameters.

[0076] FIG. 5 is a flowchart of a deep learning recognition data generation method according to one embodiment.

[0077] According to one embodiment, the following operations 510 and 520 are performed after operation 410 described above with reference to Figure 4 is performed. Operations 510 and 520 may be performed by an electronic device (e.g., electronic device 110 of Figures 1 and 2).

[0078] According to one embodiment, operation 510 is performed by a deep learning unit (e.g., deep learning unit 182-1 in FIG. 2) of a deep learning recognition unit (e.g., deep learning recognition unit 182 in FIG. 1 and FIG. 2) of an electronic device, and operation 520 is performed by a data processing unit (e.g., data processing unit 182-2 in FIG. 2) of the deep learning recognition unit.

[0079] In operation 510, the deep learning unit determines target object information for the first processing data using the deep learning video recognition model. The target object information may include at least one of the target object's type, coordinates, shape, or a score indicating recognition accuracy. For example, the target object's type may include a person, a vehicle, a bicycle, or other road / path installation. For example, the target object's shape may include the target object's width, height, etc.

[0080] According to an embodiment, the target object refers to an object that may cause a dangerous situation for the vehicle among various objects recognized by the deep learning image recognition model. For example, the deep learning unit may determine an object moving straight in the direction of travel of the vehicle or an object located on the travel path as the target object. Depending on the embodiment, a series of processes of detecting an object and determining the target object may be handled as one process or processed all at once.

[0081] According to an embodiment, if the score of the target object is less than a predetermined first threshold, the deep learning unit may not transmit the target object information to the data processing unit. Here, the data processing unit may recognize the score of the target object as 0. If the score of the target object is equal to or greater than the predetermined first threshold, the deep learning unit may transmit the target object information to the data processing unit.

[0082] In operation 520, the data processing unit can generate deep learning recognition data by determining whether a target object has been detected in a digital image corresponding to a current frame based on previous object information and target object information for previous frames within a predetermined frame range.

[0083] According to an embodiment, the data processing unit may determine that a target object is detected in the digital image if the score for the target object in the current frame is less than a predetermined first threshold and the average value of previous scores for the target object in previous frames is equal to or greater than a predetermined second threshold. The first threshold and the second threshold may be the same or different. The average value of the scores of the previous frames may represent an arithmetic average or a weighted average. For example, an average value is calculated in which a higher weight is applied to a previous frame closer to the current frame.

[0084] For illustrative purposes, assume that the first and second predetermined thresholds are 60%. For example, if the score for a target object (e.g., a person or a bicycle) in a current frame is 80%, the data processor may determine detection of the target object in the digital image. For example, if the score for a target object in a current frame is 50% and the average value of previous scores for the same target object in previous frames is 70%, the data processor may determine detection of the target object in the digital image.

[0085] According to one embodiment, the data processing unit can determine coordinates for the target object in the current frame based on the coordinates of the target object for previous frames within a predetermined frame range (or an average value of the coordinates for the previous frames).

[0086] FIG. 6 is a flowchart of a synthetic video generating method according to an embodiment.

[0087] According to one embodiment, the following operations 610 and 620 are performed after operation 340 described above with reference to Figure 3 is performed. Operations 610 and 620 may be performed by a compositor (e.g., compositor 192 of Figure 2) of an electronic device (e.g., electronic device 110 of Figures 1 and 2).

[0088] In operation 610, the synthesis unit generates a marker image layer as a computer graphic, which displays the position of the target object, based on the target object information included in the deep learning recognition data and the correction information included in the computer recognition data. The synthesis unit can generate a marker image layer as a computer graphic, which displays the position of the target object, using the coordinates of the target object and the correction information.

[0089] In operation 620, the compositor generates a composite image by combining the digital image and the computer graphics. For example, the compositor may combine the digital image and the computer graphics by overlapping them.

[0090] FIG. 7 is a diagram illustrating a composite image according to an example.

[0091] 7 shows a screen in which a composite image 700 is output through a display of an analog signal receiving device (e.g., analog signal receiving device 130 in FIG. 1) while reversing when an electronic device (e.g., electronic device 110 in FIG. 1 and FIG. 2) is connected to a target camera that captures the rear of the vehicle. For example, some (e.g., trees) of recognized objects (e.g., trees, other vehicles, people, guardrails, etc.) may not be determined as target objects, and a person running in the direction of travel of the vehicle may be determined as target object 710. A synthesis unit (e.g., synthesis unit 192 in FIG. 1 and FIG. 2) of the electronic device may generate the composite image 700 by synthesizing computer graphics 720 for the target object 710 with a digital image captured of the rear of the vehicle.

[0092] The synthesis unit may generate a marker image layer as computer graphic 720 that displays the position of the target object 710 based on the target object information included in the deep learning recognition data and the correction information included in the computer vision recognition data. The synthesis unit may also generate the marker image layer as computer graphic 720 based on the re-determined target object information included in the computer vision recognition data. The synthesis unit may generate the synthetic image 700 by synthesizing the digital image and the computer graphic 720. For example, the synthesis unit may generate the synthetic image 700 in which a frame (or box) marker is displayed at the position of the target object 710. The synthetic image 700 may be generated such that the computer graphic 720 moves together with the movement of the target object 710 to display the position of the target object 710.

[0093] The information contained in the composite image 700 is not limited to the example shown in FIG. 7, and the synthesis unit can generate the composite image 700 so that the signal input status from any external device of the vehicle, the vehicle's position information, etc. are shown.

[0094] FIG. 8 is a flowchart illustrating reception of operation information according to an embodiment.

[0095] According to one embodiment, operations 380 of Figure 3 include the following operations 810 and 820. Operations 810 and 820 may be performed in whole or in part by an electronic device (eg, electronic device 110 of Figures 1 and 2).

[0096] According to one embodiment, the vehicle operation information includes vehicle motion information and position information.

[0097] In operation 810, the electronic device receives vehicle operation information from a travel information relay module (e.g., the travel information relay module 142 in FIG. 1). The vehicle operation information may include one or more of the following: left / right turn signal lamp information, reverse gear information, brake information, or speed information of the vehicle. The travel information relay module may obtain information on whether the turn signal lamp is on, whether the reverse gear is changed, or the operation state of the brake pedal based on an analog signal that can be obtained through the power cable of the turn signal lamp, reverse lamp, or brake lamp of the vehicle. If the travel information relay module is not connected to the GPS module, the travel information relay module can obtain the speed information through a power cable that is connected to a speed sensor that measures the vehicle speed by counting the vehicle rotation number of the vehicle. If the travel information relay module is connected to the GPS module, the travel information relay module can obtain the speed information by calculating the travel distance and time interval of the vehicle using the position information received from the GPS module.

[0098] In operation 820, the electronic device receives vehicle location information from a GPS module (e.g., GPS module 144 of FIG. 1). The electronic device may receive vehicle location information from the GPS module directly or via a traffic information relay module.

[0099] According to one embodiment, the electronic device can calculate the speed by calculating the distance and time interval of the vehicle using the location information. However, when the GPS module has difficulty receiving a signal normally, such as when passing through a tunnel, the operation information relay module may obtain the speed information through a power cable of the vehicle's speed sensor.

[0100] FIG. 9 is a flowchart of a danger warning output method according to an embodiment.

[0101] According to one embodiment, the following operation 910 is performed after operation 390 of Figure 3. Operation 910 is performed by an electronic device (eg, electronic device 110 of Figures 1 and 2).

[0102] In operation 910, the electronic device outputs a danger alarm when a danger element is present. For example, the electronic device may generate a danger alarm sound when a danger element is present. For example, the electronic device may output a pre-stored sound or alarm via a speaker or buzzer.

[0103] FIG. 10 is a block diagram of an electronic device according to an embodiment.

[0104] An electronic device 1000 according to an embodiment of the present invention is similar to the electronic device 110 in Fig. 1 and Fig. 2. The electronic device 1000 includes a communication unit 1010, a memory 1020, a processor 1030, and an audio output unit 1040.

[0105] The communication unit 1010 is connected to a vehicle, a camera module provided in the vehicle, an external device, and an analog signal receiving device to transmit and receive data. The communication unit 1010 may be connected to another external device to transmit and receive data. In the following, the expression of transmitting and receiving "A" indicates transmitting and receiving "information or data indicating A".

[0106] The communication unit 1010 may be realized in a circuit network within the electronic device 1000. For example, the communication unit 1010 includes an internal bus and an external bus. For example, the communication unit 1010 may be an element that connects the electronic device 1000 and an external device. The communication unit 1010 may be an interface. The communication unit 1010 receives data from an external device (for example, a vehicle, an operation information relay module) and transmits the data to the processor 1030 and the memory 1020.

[0107] The memory 1020 can store data processed by the processor 1030 and data to be processed as hardware for storing various data processed in the electronic device 1000. The memory 1020 may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0108] The processor 1030 controls the overall operation of the electronic device 1000. The processor 1030 may execute a set of instructions (e.g., software) stored in the memory 1020 to operate the electronic device 1000. The processor 1030 may be embodied as an array of multiple logic gates, or as a combination of a general-purpose microprocessor and a memory storing a program to be implemented by the microprocessor. Those skilled in the art will understand that the processor 1030 may also be embodied in other forms of hardware.

[0109] When a risk factor for vehicle operation exists, the audio output unit 1040 provides audible information about the risk to the user. For example, the audio output unit 1040 may convert an electric signal into an audio signal and output the audio signal to the outside.

[0110] The embodiments described above may be implemented using hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or other devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, the processing device may be described as being one in which only one processing device is used, but one skilled in the art will appreciate that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0111] The software may include computer programs, codes, instructions, or any combination of one or more thereof, capable of configuring or instructing a processing device to operate as desired, either independently or in combination. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, either permanently or temporarily, to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed across computer systems coupled to a network, and may be stored and executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

[0112] The method according to the present invention may be embodied in the form of program instructions to be executed by various computer means and recorded on a computer-readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and the program instructions may be specially designed and constructed for the purposes of the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, flash memories, and the like. Examples of program instructions include not only machine language code, such as that generated by a compiler, but also high-level language code executed by a computer using an interpreter, etc.

[0113] The hardware devices described above may be configured to operate as one or more software modules to perform the operations illustrated in the present invention, and vice versa.

[0114] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than described, and may be replaced or substituted with other components or equivalents to achieve appropriate results.

[0115] Accordingly, other implementations, embodiments, and equivalents of the claims are intended to be within the scope of the following claims.

Claims

1. A method for outputting an analog signal of a composite image, the method being performed by an electronic device provided in a vehicle, comprising: receiving an analog signal for an original image captured using a target camera mounted on a vehicle; converting the analog signal for the original image into a digital signal based on a resolution of the original image; generating a digital image based on the digital signal; determining a target object within the digital image using one or more image recognition models; generating a composite image by combining a computer graphic of the target object with the digital image; converting the composite image into an analog signal; transmitting the analog signal for the composite image to an analog signal receiving device provided in the vehicle; A method for outputting an analog signal of a composite image, comprising:

2. The method of claim 1 , wherein the operation of generating a digital image based on the digital signal includes an operation of generating the digital image having a format of raw data or YUV data based on the digital signal.

3. The operation of determining a target object within the digital image using the one or more image recognition models comprises: generating first processing data for a deep learning image recognition method and second processing data for a computer vision image recognition method based on the digital image; generating deep learning recognition data for the first processing data using a deep learning image recognition model; generating computer vision recognition data for the second processed data using a computer vision image recognition model; determining a target object within the digital image based on the deep learning recognition data and the computer vision recognition data; 2. The method for outputting a composite video analog signal according to claim 1, comprising:

4. 4. The method of claim 3, wherein the operation of generating the first processed data and the second processed data based on the digital image includes an operation of generating the first processed data and the second processed data by applying at least one process of color format conversion, filtering, noise reduction, cropping, or scaling to the digital image.

5. The operation of generating the deep learning recognition data for the first processing data using the deep learning image recognition model includes: determining target object information for the first processed data using the deep learning image recognition model; determining whether the target object is detected in the digital image corresponding to a current frame based on previous object information and the target object information for a previous frame within a predetermined frame range, thereby generating the deep learning recognition data; 4. The method for outputting a composite video analog signal according to claim 3, comprising:

6. The method of claim 5 , wherein the target object information includes at least one of a type, a coordinate, a shape, or a score indicating a recognition accuracy of the target object.

7. The method of claim 6, wherein the operation of generating the deep learning recognition data by determining whether the target object is detected in the digital image includes an operation of determining whether the target object is detected in the digital image if the score for the target object in the current frame is less than a predetermined first threshold and an average value of previous scores for the target object in the previous frames is greater than or equal to a predetermined second threshold.

8. The method for outputting an analog signal of a synthetic image as described in claim 3, wherein the operation of generating computer vision recognition data for the second processed data using the computer vision image recognition model includes an operation of generating the computer vision recognition data for the second processed data using the computer vision image recognition model, the computer vision recognition data including correction information based on camera parameters for the camera.

9. The operation of generating a composite image by synthesizing a computer graphic of the target object with the digital image includes an operation of generating a marker image layer as the computer graphic, the marker image layer displaying the position of the target object based on target object information included in the deep learning recognition data and correction information included in the computer recognition data; generating the composite image by combining the digital image and the computer graphic; 4. The method for outputting a composite video analog signal according to claim 3, comprising:

10. 2. The method for outputting an analog signal of a composite video signal according to claim 1, wherein the analog signal receiving device is a black box module.

11. receiving operation information of the vehicle; An operation of determining whether or not there is a risk factor for the current state of the vehicle based on the vehicle operation information and the deep learning recognition data; The method for outputting a composite video analog signal according to claim 3, further comprising:

12. The method for outputting an analog signal of a composite image as described in claim 11, wherein the operation of receiving the vehicle's operation information may include at least one of an operation of receiving the vehicle's operation information from an operation information relay module or an operation of receiving the vehicle's position information from a GPS module.

13. The method of claim 11, further comprising the step of outputting a danger alarm when the danger element exists.

14. A computer-readable recording medium storing a program for executing the method according to any one of claims 1 to 13.

15. 1. An electronic device for analog signal output of a composite image, comprising: a first conversion unit that receives an analog signal corresponding to an original image captured by a target camera mounted on a vehicle; a signal processing unit for converting the analog signal for the original image into a digital signal based on a resolution of the original image; an image processing unit that generates a digital image based on the digital signal; a video recognition unit for determining target objects within the digital video using one or more video recognition models; a hazard detection software unit that generates a composite image by combining a computer graphic of the target object with the digital image, converts the composite image into an analog signal, and transmits the analog signal for the composite image to an analog signal receiving device provided in the vehicle; 23. An electronic device comprising:

16. The image recognition unit is generating first processing data for a deep learning image recognition method and second processing data for a computer vision image recognition method based on the digital image; generating deep learning recognition data for the first processing data using a deep learning image recognition model; generating computer vision recognition data for the second processed data using a computer vision image recognition model; determining a target object within the digital image based on the deep learning recognition data and the computer vision recognition data; The electronic device according to claim 15 .

17. The operation of generating the deep learning recognition data for the first processing data using the deep learning image recognition model includes: determining target object information for the first processed data using the deep learning image recognition model; determining whether the target object is detected in the digital image corresponding to a current frame based on previous object information and the target object information for a previous frame within a predetermined frame range, thereby generating the deep learning recognition data; 17. The electronic device of claim 16, comprising:

18. The electronic device of claim 17 , wherein the target object information may include at least one of a type, coordinates, shape, or a score indicating a recognition accuracy of the target object.

19. 20. The electronic device of claim 18, wherein the operation of generating the deep learning recognition data by determining whether the target object is detected in the digital image includes an operation of determining whether the target object is detected in the digital image if the score for the target object in the current frame is less than a predetermined first threshold and an average value of previous scores for the target object in the previous frames is greater than or equal to a predetermined second threshold.

20. The operation of generating a composite image by compositing a computer graphic of the target object with the digital image includes: An operation of generating a marker image layer that displays the position of the target object as the computer graphic based on target object information included in the deep learning recognition data and correction information included in the computer recognition data; generating the composite image by combining the digital image and the computer graphic; 17. The electronic device of claim 16, comprising:

Citation Information

Patent Citations

  • Video telephone set

    JP1994197335A

  • Measuring method and measuring device for vehicular distance based on vehicle video, electronic apparatus, computer program and computer readable recording medium

    JP2021118002A

  • Learning device, learning method, and learning program

    JP7236062B2