An image sensor and an electronic rearview mirror

Abstract

This application discloses an image sensor and an electronic rearview mirror, relating to the field of image processing. The image sensor is deployed in an image acquisition device, which is connected to a controller, and the controller is connected to a display. The image sensor includes at least: a sensor core module for converting acquired light signals into digital electrical signals; an image signal processing module for performing signal optimization processing on the digital electrical signals; a signal encoding module for encoding the optimized digital electrical signals to obtain image data; and a data interface for transmitting the image data to the controller. The controller processes the image data and transmits the obtained target image to the display, which then outputs the target image.

Description

Technical Field

[0001] This application relates to the field of electronic rearview mirror technology, and more particularly to an image sensor and an electronic rearview mirror. Background Technology

[0002] An electronic rearview mirror (CMS) is a system composed of a camera and a monitor. Within a specified field of view, it is required to clearly see the area behind and to the sides of the vehicle, representing a new type of indirect vision device. The advantages of electronic rearview mirrors include eliminating the field-of-view limitations imposed by traditional optical rearview mirrors due to factors such as mirror curvature, shape, size, and angle. The overall field of view is wider, and visibility in low-light conditions is effectively improved, allowing the driver to see more details and enhancing driving safety.

[0003] However, the signal encoding module in the electronic rearview mirror is deployed on the printed circuit board where the image sensor is located. The data transmission and data encoding on the board have a large delay, which results in a large delay in the image output of the electronic rearview mirror. Summary of the Invention

[0004] In view of the above problems, this application provides an image sensor and an electronic rearview mirror to improve the image output efficiency of the electronic rearview mirror. The specific solution is as follows:

[0005] The first aspect of this application provides an image sensor deployed in an image acquisition device, the image acquisition device being connected to a controller, and the controller being connected to a display.

[0006] The image sensor includes at least:

[0007] The sensor core module is used to convert the acquired optical signals into digital electrical signals;

[0008] An image signal processing module is used to perform signal optimization processing on the digital electrical signal;

[0009] A signal encoding module is used to encode the optimized digital electrical signal to obtain image data;

[0010] A data interface is used to transmit the image data to the controller, the controller processes the image data and transmits the resulting target image to the display, and the display outputs the target image.

[0011] In one possible implementation, the signal encoding module includes: a serializer module; the image data output by the serializer module is a digital differential signal; the serializer module includes: a data input interface, a control and data conversion module, and a serial interface;

[0012] The data input interface is used to transmit the digital electrical signal from the image signal processing module to the control and data conversion module;

[0013] The control and data conversion module is used to encode the digital electrical signal to obtain image data;

[0014] The serial interface is used to transmit the image data to the data interface.

[0015] In one possible implementation, the signal encoding module includes: an optical communication module; the image data output by the optical communication module is an optical signal; the optical communication module includes: a data transmission interface, a data codec, an optical signal processor, an optical communication interface, and a control module;

[0016] The data transmission interface is used to transmit the digital electrical signal from the image signal processing module to the data codec.

[0017] The data codec is used to encode the digital electrical signal using an algorithm, and transmit the encoded image data to the optical signal processor.

[0018] The optical signal processor is used to convert the image data encoded by the data codec into optical signals;

[0019] The optical communication interface is used to transmit image data from the optical signal processor to the data interface.

[0020] In one possible implementation, the image sensor is deployed as a pixel layer, a logic circuit layer, and a cache layer; the sensor kernel module is deployed as a signal conversion module in the pixel layer and the logic circuit layer; the image signal processing module is deployed as an image signal processor in the logic circuit layer; wherein:

[0021] The pixel layer is used to collect optical signals and convert the optical signals into analog charge signals;

[0022] The signal conversion module of the logic circuit layer is used to convert the analog charge signal into a digital electrical signal;

[0023] The image signal processor in the logic circuit layer is used to perform signal optimization processing on the digital electrical signal;

[0024] The cache layer is used to store the digital electrical signals output by the logic circuit layer; the logic circuit layer reads the digital electrical signals from the cache layer and transmits them to the signal encoding module.

[0025] In one possible implementation, when the logic circuit layer converts the analog charge signal transmitted from the pixel layer into a digital electrical signal and transmits it to the signal encoding module, the image data obtained by the signal encoding module through data encoding of the digital electrical signal is the first frame rate.

[0026] When the logic circuit layer reads the digital electrical signal from the cache layer and transmits it to the signal encoding module, the image data obtained by the signal encoding module through data encoding of the digital electrical signal is the second frame rate;

[0027] The second frame rate is greater than the first frame rate.

[0028] A second aspect of this application provides an electronic rearview mirror, comprising:

[0029] The system includes a controller, at least one display, and at least one image acquisition device, wherein the image acquisition device is connected to the controller, and the controller is connected to the display.

[0030] The image acquisition device includes at least one of the following: an image sensor as described in any of the above.

[0031] In one possible implementation, the controller includes at least: an in-memory computing accelerator and a signal decoding module;

[0032] The signal decoding module is used to decode the image data obtained by the image acquisition device through data encoding and transmit the decoded image data to the in-memory computing accelerator.

[0033] The in-memory computing accelerator performs image processing on the image data transmitted by the signal decoding module to obtain an intermediate image, and stitches at least two of the intermediate images together to obtain the target image.

[0034] The in-memory computing accelerator includes: an image processor and an image stitching module;

[0035] The image processor is used to perform the image processing computation task; the image stitching module is used to perform the image stitching computation task; the image processor and the image stitching module perform computation in parallel.

[0036] In one possible implementation, the in-memory computing accelerator includes at least: an in-memory computing unit; the in-memory computing unit performs the image processing computing task and the image stitching computing task through multiple computing units and a weight storage unit;

[0037] The weight storage unit is used to store the image data, and the plurality of computing units update the feature data in the image data in the weight storage unit.

[0038] In one possible implementation, the plurality of computing units perform the image processing computation task and the image stitching computation task through a deployed intelligent model.

[0039] In one possible implementation, the in-memory computing accelerator further includes:

[0040] Accumulator, pooler, multiply-accumulator, quantizer, activator, activation value buffer;

[0041] The accumulator is used to accumulate the data output by the in-memory computing unit; the pooler is used to pool the data output by the accumulator; the multiply-accumulate is used to multiply-accumulate the data output by the pooler; the quantizer is used to quantize the data output by the multiply-accumulate; the activator is used to activate the data output by the quantizer; and the activation value buffer is used to store the activation value corresponding to the activator and provide the activation value to the in-memory computing unit.

[0042] In one possible implementation, the activation value cache performs full-duplex data transmission with the in-memory computing unit via a dual-cache mode.

[0043] In one possible implementation, the controller further includes a serializer; the display includes a deserializer and a display panel; the serializer is connected to the deserializer;

[0044] The serializer converts the target image obtained by the in-memory computing accelerator into serial data and transmits the serial data to the deserializer. The deserializer converts the serial data into parallel data and transmits the parallel data to the display panel. The display panel refreshes the screen based on the parallel data.

[0045] By means of the above technical solution, in the image sensor and electronic rearview mirror provided in this application, the signal encoding module is changed from being outside the image sensor to being deployed inside the image sensor. This enables image data transmission and data encoding to be realized within the chip of the image sensor. In turn, through chip-level data encoding and transmission, the output efficiency of the image acquisition device can be accelerated, thereby improving the image output efficiency of the electronic rearview mirror. Attached Figure Description

[0046] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1 This is a diagram showing the output delay of an electronic rearview mirror in the prior art.

[0048] Figure 2 This is a schematic diagram of the structure of an image sensor provided in an embodiment of this application;

[0049] Figure 3 This is a structural diagram of the electronic rearview mirror in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram illustrating the deployment of the electronic rearview mirror in an embodiment of this application;

[0051] Figure 5 This application provides a schematic diagram of the structure of a serializer module in an image sensor.

[0052] Figure 6 This is a schematic diagram of the image sensor structure with an integrated serializer module in an embodiment of this application;

[0053] Figure 7 This application provides a schematic diagram of the structure of an optical communication module in an image sensor.

[0054] Figure 8 This is a schematic diagram of the image sensor with an integrated optical communication module in an embodiment of this application;

[0055] Figure 9 This is an example diagram showing the image sensor deployed in an image acquisition device according to an embodiment of this application;

[0056] Figure 10 This is a storage diagram of the parameter storage unit in an embodiment of this application;

[0057] Figure 11 This is a schematic diagram of the deployment architecture layer of the image sensor in the embodiments of this application;

[0058] Figure 12 This is a schematic diagram illustrating the data interaction between the cache layer and the ultra-high frame rate in an embodiment of this application.

[0059] Figure 13 This is a schematic diagram of the structure of an electronic rearview mirror provided in an embodiment of this application;

[0060] Figure 14 A schematic diagram of the structure of a controller implemented in a SoC in an electronic rearview mirror provided in an embodiment of this application;

[0061] Figure 15 A schematic diagram of the structure of an in-memory computing accelerator in the controller of an electronic rearview mirror provided in an embodiment of this application;

[0062] Figure 16 This is a schematic diagram of the structure of the in-memory computing unit in the SoC in the embodiments of this application;

[0063] Figure 17 A schematic diagram illustrating the computational tasks performed by a conventional image processor (ISP) in existing technology;

[0064] Figure 18 This is a schematic diagram illustrating image stitching using a traditional image stitching module in the prior art.

[0065] Figure 19 This is a schematic diagram of the structure of an electronic rearview mirror provided in an embodiment of this application;

[0066] Figure 20 This is an example diagram of an electronic rearview mirror in an embodiment of this application;

[0067] Figure 21 This is an example diagram of a camera in an embodiment of this application;

[0068] Figure 22 This is a block diagram showing the image output delay of the electronic rearview mirror in an embodiment of this application. Detailed Implementation

[0069] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0070] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0071] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0072] For CMS (Content Management System), industry standards stipulate that the image output latency should be sufficiently short, less than 200ms at room temperature (22℃±5℃). However, in practical applications, a latency of 200ms is far from sufficient; currently, CMS latency commonly exceeds 50ms, representing a pressing technical challenge for the industry. Typically, CMS image output latency includes the imaging latency of image acquisition devices such as cameras, the processing latency of the System-on-a-Chip (SoC), and the screen display latency. Camera imaging latency and screen display latency each represent the time required to capture one frame. SoC processing latency includes image data input buffering time, image data processing time, image stitching time, and data output buffering time; SoC processing latency generally exceeds 30ms.

[0073] like Figure 1 As shown, the calculation of the plotting delay is as shown in formula (1), T 系统 T1 is the system processing time, T2 is the camera imaging time, T3 is the SoC processing time, and T4 is the display time.

[0074] T 系统 =T1+T2+T3(1)

[0075] Taking a 60-frame image as an example, the image output delay will exceed 50ms, as shown in formula (2):

[0076] T 系统 =T1+T2+T3=16.67ms+30ms+16.67ms=63.33ms (2)

[0077] To address the aforementioned problems, embodiments of this application provide an image sensor, an image acquisition device, and an electronic rearview mirror. The image sensor, image acquisition device, and electronic rearview mirror of this application will be described in detail below with reference to the accompanying drawings.

[0078] Reference Figure 2 This is a schematic diagram of the structure of an image sensor provided in an embodiment of this application. The image sensor can be deployed in any type of image acquisition device. The image acquisition device is connected to a controller, and the controller is connected to a display. The image acquisition device can be deployed in any device that requires image acquisition, such as an electronic rearview mirror. For example, as... Figure 3 As shown, at least one image acquisition device equipped with the image sensor of this embodiment is connected to a controller, and the controller is connected to at least one display. The image acquisition device, controller, and display can form an electronic rearview mirror. The technical solution in this embodiment is mainly used to improve the image sensor's efficiency in acquiring image data.

[0079] Specifically, the image sensor in this embodiment may include at least the following structure:

[0080] The sensor core module 201 is used to convert the acquired optical signals into digital electrical signals;

[0081] Image signal processing module 202 is connected to sensor core module 201 and is used to perform signal optimization processing on digital electrical signals.

[0082] The signal encoding module 203 is connected to the image signal processing module 202. The signal encoding module 203 is used to encode the optimized digital electrical signal to obtain image data.

[0083] Data interface 204 is connected to signal encoding module 203. Data interface 204 is used to transmit image data to the controller connected to the image acquisition device. The controller processes the image data and transmits the obtained target image to the display, and the display outputs the target image.

[0084] It should be noted that the sensor core module 201, image signal processing module 202, signal encoding module 203 and data interface 204 are integrated within the image sensor implementation chip, and data transmission between the modules is achieved within the chip.

[0085] In practice, the image acquisition device can be a camera, such as a Category 2, Category 4, or Category 6 camera. Different types of cameras use different image sensors, acquire different image resolutions, perform different functions, and have different application scenarios. For example, the image resolution of a Category 2 camera is lower than that of a Category 4 camera, and vice versa. The image acquisition device collects image data and sends it to the controller. Based on this, the controller processes the image data, performing actions such as image quality optimization, distortion correction, scaling, and image stitching, and then transmits the resulting target image to the display screen, which outputs the target image.

[0086] For example, take a vehicle's electronic rearview mirror as an example, such as Figure 3 As shown, the vehicle is equipped with two Category 2 rearview mirror cameras, two Category 4 rearview mirror cameras, and one Category 6 rearview mirror camera. Each camera is connected to a controller, which in turn is connected to two displays. For example, as... Figure 4As shown in the diagram of the vehicle's front and interior, the vehicle has a Category 2 camera on each of the left and right sides of the front, a Category 4 camera on each of the left and right sides of the front, and a Category 6 camera at the front of the vehicle. A controller (an integrated chip containing multiple control chips or a single control chip) is located inside the front, and a display is located on each of the left and right sides inside the front. Based on this, each camera in the vehicle outputs its acquired image data to the controller, which processes the image data and transmits the resulting target image to the two displays. The images output by both displays are either the target image provided by the controller, or the two displays output the left and right portions of the target image respectively.

[0087] As can be seen from the above scheme, in the image sensor provided in this application embodiment, the signal encoding module is deployed inside the image sensor. In this way, the signal encoding module and the image signal processing module inside the image sensor are connected on-chip, which can realize chip-level data transmission and data encoding. Compared with the signal encoding module being deployed on the printed circuit board where the image sensor is located to realize on-board data transmission and data encoding, chip-level data encoding and data transmission can accelerate the image sensor's image data acquisition efficiency, thereby improving the image data output efficiency of the image acquisition device where the image sensor is located, and thus improving the image output efficiency of the electronic rearview mirror where the image acquisition device is located.

[0088] In one implementation, the signal encoding module 203 in the image sensor can be a serializer module or an optical communication module;

[0089] The image data output by the serializer module is a digital differential signal, while the image data output by the optical communication module is an optical signal.

[0090] Based on this, in this embodiment, the signal encoding module 203 in the image sensor can be determined to be a serializer module or an optical communication module based on business requirements or the application scenario of the image sensor. Thus, the image sensor in this embodiment can be applied to more application scenarios and meet the user's needs for the image sensor.

[0091] Based on the above implementation scheme, the serializer module may include: a data input interface, a clock management module, a control and data conversion module, a serial interface, and an input / output (IO) interface.

[0092] For example, such as Figure 5 As shown, the data input interface is used to transmit the digital electrical signals from the image signal processing module 202 to the control and data conversion module. The data input interface supports the input of parallel data (signals) and serial data.

[0093] The control and data conversion module is used to control the working logic of the serializer module, and to encode (convert) digital electrical signals to obtain image data.

[0094] The serial interface is used to transmit the encoded image data to the data interface 204 at high speed.

[0095] The clock management module is used to provide a clock source for each module in the serializer module.

[0096] The IO interface is the peripheral interface of the serializer module, used to connect to other modules.

[0097] Based on this, the serializer module passes through Figure 5 Each module shown implements data encoding and high-speed data transmission of digital electrical signals. The transmitted image data is a digital differential signal that can be transmitted at high speed.

[0098] As can be seen, in this embodiment, the generation and transmission of digital differential signal image data can be achieved through the serializer module, which can meet the user's needs for image sensor.

[0099] For example, image sensors with integrated serializer modules, such as Figure 6 As shown, the image sensor, in addition to including a sensor core module 201, an image signal processing module 202, a signal encoding module 203 (i.e., a serializer module), and a data interface 204, may also include a control unit 205 and a clock unit 206, wherein:

[0100] The sensor kernel module 201 is used to collect optical signals and convert them into analog charge signals, and then convert the analog charge signals into digital electrical signals.

[0101] The image signal processing module 202 is used to perform signal optimization processing on digital electrical signals.

[0102] The serializer module (i.e., signal encoding module 203) is used to serially encode digital electrical signals to obtain image data.

[0103] Data interface 204 (serial interface in the serializer module) is used to interact with other modules outside the image sensor, such as controllers.

[0104] The control unit 205 is used to control the workflow of the image sensor.

[0105] The clock unit 206 is used to provide a clock source for each module in the image sensor.

[0106] Therefore, in this embodiment, the generation and transmission of digital differential signal image data can be achieved through the serializer module, which can meet the user's needs for image sensor.

[0107] Based on the above implementation scheme, the optical communication module may include: data transmission interface, data codec, optical signal processor, optical communication interface and control module, etc.

[0108] For example, such as Figure 7 As shown, the data transmission interface can be a bidirectional channel for transmitting digital electrical signals from the image signal processing module 202 to the data codec; or, the data transmission interface can also be used to transmit digital data decoded by the data codec, such as control signals, to other modules in the image sensor, such as the control unit in the image sensor.

[0109] The data codec consists of a data encoder and a data decoder. The data encoder is used to encode digital electrical signals using algorithms, and then transmits the encoded image data to the optical signal processor; while the data decoder is used to decode digital data, such as control signals, transmitted from the optical signal processing module and transmit them to the data transmission interface.

[0110] An optical signal processor is used to convert image data (digital signals) encoded by a data encoder into optical signals; or, an optical signal processing module converts optical signals, such as control signals, transmitted from an optical communication interface into digital data.

[0111] The optical communication interface is used to transmit image data (optical signals) from the optical signal processor to the data interface 204 (provided to the controller), or to receive control signals (optical signals) sent by the data interface 204 (obtained from the controller).

[0112] The control module is responsible for the working logic control of the entire optical communication module and provides the clock source.

[0113] Based on this, the optical communication module passes through Figure 7 Each module shown realizes the data encoding of digital electrical signals and the conversion of optical signals, as well as high-speed data transmission. The transmitted image data is an optical signal that can be transmitted at high speed.

[0114] As can be seen, in this embodiment, the generation and transmission of optical signal image data can be achieved through the serializer module, which can meet the user's needs for using the image sensor.

[0115] For example, image sensors with integrated optical communication modules, such as Figure 8 As shown, the image sensor, in addition to including a sensor core module 201, an image signal processing module 202, a signal encoding module 203 (i.e., an optical communication module), and a data interface 204, may also include a control unit 205 and a clock unit 206, wherein:

[0116] The sensor kernel module 201 is used to collect optical signals and convert them into analog charge signals, and then convert the analog charge signals into digital electrical signals.

[0117] The image signal processing module 202 is used to perform signal optimization processing on digital electrical signals.

[0118] The optical communication module (i.e., signal encoding module 203) is used to optically encode digital electrical signals to obtain image data.

[0119] Data interface 204 (optical communication interface for connecting optical communication module) is used to interact with other modules outside the image sensor, such as controller.

[0120] The control unit 205 is used to control the workflow of the image sensor.

[0121] The clock unit 206 is used to provide a clock source for each module in the image sensor.

[0122] Therefore, in this embodiment, the optical communication module can be used to generate and transmit image data of optical signals, which can meet the user's needs for image sensors.

[0123] In one implementation, the image sensor in this embodiment is connected to a parameter storage unit, which is deployed in an image acquisition device. The image acquisition device also includes a data interface for transmitting image data output by the image sensor to a controller connected to the image acquisition device. The controller is connected to a display, such as... Figure 9 As shown, the parameter storage unit stores at least the startup parameters, which are used to start the image acquisition device where the image sensor is located.

[0124] Based on this, the image sensor may also include a control unit 205, such as Figure 9 As shown, the control unit 205 is connected to the sensor kernel module 201, the image signal processing module 202, and the signal encoding module 203, respectively. The control unit 205 is also connected to the parameter storage unit. Thus, the control unit 205 can read the start parameters from the parameter storage unit in response to the start command of the image acquisition device and set the corresponding configuration parameters of the sensor kernel module 201, the image signal processing module 202, and the signal encoding module 203 according to the start parameters. The sensor kernel module 201, the image signal processing module 202, and the signal encoding module 203 operate according to their respective set configuration parameters, thereby enabling the image sensor to output image data.

[0125] The start command for the image acquisition device can be a command generated when the image acquisition device is powered on. The start command is used to instruct the image acquisition device to acquire image data.

[0126] As can be seen, in this embodiment, the startup parameters are stored in the parameter storage unit connected to the image sensor within the image acquisition device. Therefore, when the image acquisition device is powered on, the controller connected to the image acquisition device does not need to set the configuration parameters of each module of the image sensor. This eliminates the need for the controller to participate in reading, writing, and setting the startup parameters. Instead, the control unit within the image sensor directly reads the startup parameters from the parameter storage unit deployed in the image acquisition device and sets the configuration parameters of the corresponding modules. This achieves chip-level data reading, writing, and parameter setting, thereby accelerating the configuration parameter setting of each module in the image sensor, thus speeding up the image sensor's startup process. This, in turn, accelerates the image sensor's light signal acquisition, signal conversion, signal optimization, and data encoding, thereby increasing the output efficiency of image data in the image sensor. Ultimately, this improves the image output efficiency of the image acquisition device and improves the image output efficiency of the electronic rearview mirror.

[0127] Furthermore, the parameter storage unit can also store system parameters and user parameters, such as... Figure 10 As shown, the startup parameters are stored at the starting address of the parameter storage unit.

[0128] As can be seen, in this embodiment, the storage management of the image sensor’s startup parameters is optimized in the parameter storage unit. The minimum parameters required for the image sensor to start and output images (i.e., startup parameters) are fixed to the starting address of the parameter storage unit. In this way, the control unit 205 in the image sensor can read the startup parameters in the parameter storage unit at the fastest speed. After the image acquisition device is powered on, it can complete the configuration and output of the image sensor in the shortest time.

[0129] System parameters can include parameters such as time and device model; user parameters can include parameters configured based on user operations and parameters that identify the user.

[0130] Specifically, after setting the configuration parameters of the sensor kernel module 201, image signal processing module 202, and signal encoding module 203 according to the startup parameters, the control unit 205 reads the system parameters and user parameters from the parameter storage unit and sets the configuration parameters of the sensor kernel module 201, image signal processing module 202, and signal encoding module 203 according to the system parameters and user parameters; and the sensor kernel module 201, image signal processing module 202, and signal encoding module 203 operate according to the set configuration parameters.

[0131] As can be seen, in this embodiment, after the control unit in the image sensor writes the startup parameters into the corresponding module of the image sensor, it also writes the system parameters and user parameters into the corresponding module of the image sensor. This speeds up the startup process of the image sensor without affecting the related functions of the image sensor involving system parameters and user parameters.

[0132] In one implementation, the image sensor in this embodiment is deployed as a pixel layer 1101, a logic circuit layer 1102, and a cache layer 1103, such as... Figure 11 As shown, the sensor kernel module 201 is deployed as a signal conversion module in the pixel layer 1101 and the logic circuit layer 1102; the image signal processing module 202 is deployed as an image signal processor (ISP) in the logic circuit layer 1102; wherein:

[0133] The pixel layer 1101, also known as the photosensitive layer, is used to collect light signals and convert them into analog charge signals. The pixel layer 1101 can adopt back-illuminated technology, placing the circuit below the photodiode to increase the aperture ratio. In the process of photoelectric conversion of light signals to generate analog charge signals, it can improve the photosensitivity and signal-to-noise ratio.

[0134] The signal conversion module of logic circuit layer 1102 may include an analog-to-digital converter (ADC) to convert the analog charge signal transmitted from pixel layer 1101 into a digital electrical signal; the image signal processor in logic circuit layer 1102 integrates image processing algorithms to perform signal optimization processing on the digital electrical signal, including dead pixel removal, noise reduction, uniformity correction and data format conversion.

[0135] The buffer layer 1103 is used to store the digital electrical signals output by the logic circuit layer 1102; the logic circuit layer 1102 reads the digital electrical signals from the buffer layer 1103 and transmits them to the signal encoding module 203.

[0136] The cache layer 1103 can employ resistive random-access memory (RRAM) to achieve an ultra-high-speed data cache, supporting ultra-high frame rate exposure of the photosensitive layer. RRAM uses a metal oxide thin film as the storage medium, and data storage is achieved by applying pulse voltages to change the material's electrical conductivity. Compared to traditional dynamic random-access memory (DRAM), RRAM features faster read / write speeds and lower power consumption.

[0137] In addition, a control circuit module is also deployed in the logic circuit layer 1102. The control circuit module consists of a logic control circuit and a timing control circuit, which is used to control the exposure timing of the pixel layer 1101 and the reading timing of reading image data in the buffer layer 1103.

[0138] In the specific implementation, the pixel layer 1101, the logic circuit layer 1102 and the cache layer 1103 are interconnected through through-silicon via (TSV) technology, thereby achieving extremely high data transmission bandwidth.

[0139] In one implementation, when the logic circuit layer 1102 converts the analog charge signal transmitted from the pixel layer 1101 into a digital electrical signal and transmits it to the signal encoding module 203, the image data obtained by the signal encoding module 203 through data encoding of the digital electrical signal is the first frame rate.

[0140] When the logic circuit layer 1102 reads the digital electrical signal from the cache layer 1103 and transmits it to the signal encoding module 203, the image data obtained by the signal encoding module 203 through data encoding of the digital electrical signal is the second frame rate.

[0141] The second frame rate is greater than the first frame rate. For example, the second frame rate is the ultra-high frame rate, and the second frame rate is the normal frame rate.

[0142] In this specific implementation, the cache layer 1103 in this embodiment can operate in two modes: normal frame rate and ultra-high frame rate. In normal frame rate mode, when the image acquisition device, such as a camera, is in a normal scene, the data stream (analog charge signal) of the photosensitive layer (i.e., pixel layer 1101) is processed by the logic circuit layer 1102 (i.e., digital electrical signal) and directly output to the signal encoding module 203. In ultra-high frame rate mode, when the image acquisition device, such as a camera, is in a scene of rapid object movement, the massive data stream (analog charge signal) of the photosensitive layer is processed by the logic circuit layer 1102 (i.e., digital electrical signal) and quickly stored in the cache layer 1103, enabling the image sensor to achieve ultra-high frame rates, such as 960fps exposure.

[0143] For example, such as Figure 12 As shown, in a normal scene, the analog charge signal of normal frame rate is transmitted to the logic circuit layer 1102 through the pixel layer 1101, i.e. line ①. The logic circuit layer 1102 converts the analog charge signal into a digital electrical signal, and then the logic circuit layer 1102 transmits the digital electrical signal to the external signal encoding module 203, i.e. line ④. At this time, the image data obtained by the signal encoding module 203 is the normal frame rate.

[0144] In fast-moving physical scenarios, the analog charge signal with ultra-high frame rate is transmitted from the pixel layer 1101 to the logic circuit layer 1102, i.e., line ①. The logic circuit layer 1102 converts the analog charge signal into a digital electrical signal, and then stores the digital electrical signal in the cache layer 1103, i.e., line ②. After that, the logic circuit layer 1102 reads the digital electrical signal from the cache layer 1103, i.e., line ③, and then transmits the digital electrical signal to the external signal encoding module 203, i.e., line ④. At this time, the image data obtained by the signal encoding module 203 is ultra-high frame rate.

[0145] Based on this, in this embodiment, the generation process of digital electrical signals and the transmission process of digital electrical signals are isolated and decoupled through a buffer layer. In this way, the generation process of digital electrical signals is continuous and will not need to be paused when transmitting digital electrical signals to the outside. This improves the efficiency of image sensor outputting image data, thereby increasing the output rate of image data of image acquisition device, thus realizing high frame rate image output of image acquisition device, and thus improving the image output efficiency of electronic rearview mirror.

[0146] As can be seen, the sensor kernel module of the image sensor in this embodiment adopts... Figure 11 The three-layer stacked architecture shown enables efficient processing from photosensitive to signal conversion.

[0147] The above describes an image sensor provided by an embodiment of this application. The following will describe an image acquisition device that includes an image sensor.

[0148] This application also provides an image acquisition device, such as... Figure 9 As shown, the image acquisition device is connected to a controller, and the controller is connected to a display. The image acquisition device can be deployed in any device that requires image acquisition, such as an electronic rearview mirror. For example, as... Figure 3 As shown, at least one image acquisition device, such as a camera, in this embodiment is connected to a controller, and the controller is connected to at least one display. The image acquisition device, controller, and display can form an electronic rearview mirror. The technical solution in this embodiment is mainly used to improve the output efficiency of image data by the image acquisition device.

[0149] Specifically, the image acquisition device in this embodiment may include at least an image sensor, and the image sensor may be as follows: Figure 2 The diagram shows the following structures:

[0150] The sensor core module 201 is used to convert the acquired optical signals into digital electrical signals;

[0151] Image signal processing module 202 is connected to sensor core module 201 and is used to perform signal optimization processing on digital electrical signals.

[0152] The signal encoding module 203 is connected to the image signal processing module 202. The signal encoding module 203 is used to encode the optimized digital electrical signal to obtain image data.

[0153] Data interface 204 is connected to signal encoding module 203. Data interface 204 is used to transmit image data to the controller connected to the image acquisition device. The controller processes the image data and transmits the obtained target image to the display, and the display outputs the target image.

[0154] By employing the above technical solution, in the image acquisition device provided in this application embodiment, the signal encoding module is deployed within the image sensor included in the image acquisition device. In this way, the signal encoding module and the image signal processing module within the image sensor are connected within the chip, enabling chip-level data transmission and encoding. Compared to deploying the signal encoding module on the printed circuit board where the image sensor is located to achieve on-board data transmission and encoding, chip-level data encoding and transmission can accelerate the image sensor's image data acquisition efficiency, thereby improving the image data output efficiency of the image acquisition device where the image sensor is located, and thus improving the image output efficiency of the electronic rearview mirror where the image acquisition device is located.

[0155] In one implementation, the signal encoding module 203 in the image sensor of the image acquisition device can be a serializer module or an optical communication module.

[0156] The image data output by the serializer module is a digital differential signal. The module structure of the serializer module can be found by referring to... Figure 5 As shown in the image; the image data output by the optical communication module is an optical signal, and the module structure of the optical communication module can be found in [reference needed]. Figure 7 As shown in the image.

[0157] Based on this, in this embodiment, the signal encoding module 203 in the image sensor can be determined to be a serializer module or an optical communication module based on business requirements or the application scenario of the image sensor. Thus, the image sensor in this embodiment can be applied to more application scenarios and meet the user's needs for the image sensor.

[0158] In one implementation, the image acquisition device may further include: a parameter storage unit, such as... Figure 9As shown, the parameter storage unit is connected to the image sensor, which also includes a control unit 205. The control unit 205 is connected to the sensor core module 201, the image signal processing module 202, and the signal encoding module 203, respectively. The control unit 205 is also connected to the parameter storage unit. The parameter storage unit stores at least the startup parameters, which are used to start the image acquisition device.

[0159] Specifically, in response to the start command of the image acquisition device, the control unit 205 reads the start parameters from the parameter storage unit and sets the corresponding configuration parameters of the sensor kernel module 201, image signal processing module 202 and signal encoding module 203 according to the start parameters. The sensor kernel module 201, image signal processing module 202 and signal encoding module 203 operate according to their respective set configuration parameters, so that the image sensor can output image data, and correspondingly, the image acquisition device can output image data.

[0160] As can be seen, in this embodiment, the startup parameters are stored in the parameter storage unit connected to the image sensor within the image acquisition device. Therefore, when the image acquisition device is powered on, the controller connected to the image acquisition device does not need to set the configuration parameters of each module of the image sensor. This eliminates the need for the controller to participate in reading, writing, and setting the startup parameters. Instead, the control unit within the image sensor directly reads the startup parameters from the parameter storage unit deployed in the image acquisition device and sets the configuration parameters of the corresponding modules. This achieves chip-level data reading, writing, and parameter setting, thereby accelerating the configuration parameter setting of each module in the image sensor, thus speeding up the image sensor's startup process. This, in turn, accelerates the image sensor's light signal acquisition, signal conversion, signal optimization, and data encoding, thereby increasing the output efficiency of image data in the image sensor. Ultimately, this improves the image output efficiency of the image acquisition device and improves the image output efficiency of the electronic rearview mirror.

[0161] Furthermore, the parameter storage unit can also store system parameters and user parameters. System parameters may include parameters such as time and device model; user parameters may include parameters configured based on user operations and parameters that identify the user.

[0162] As can be seen, in this embodiment, after the control unit in the image sensor writes the startup parameters into the corresponding module of the image sensor, it also writes the system parameters and user parameters into the corresponding module of the image sensor. This speeds up the startup process of the image sensor without affecting the related functions of the image sensor involving system parameters and user parameters. Thus, the output efficiency of the image acquisition device for image data can be improved without affecting the corresponding functions of the image acquisition device, thereby ultimately improving the image output efficiency of the electronic rearview mirror.

[0163] In one implementation, the image sensor in this embodiment is deployed as a pixel layer 1101, a logic circuit layer 1102, and a cache layer 1103. The structure of the pixel layer 1101, the logic circuit layer 1102, and the cache layer 1103 can be referred to Figure 11 .

[0164] As can be seen, in the image acquisition device of this embodiment, the sensor kernel module in the image sensor adopts... Figure 11 The three-layer stacked architecture shown enables efficient processing from photosensitive to signal conversion.

[0165] The above describes an image acquisition device provided by an embodiment of this application. The following will describe an electronic rearview mirror that includes an image acquisition device.

[0166] refer to Figure 13 The diagram shown is a structural schematic of an electronic rearview mirror provided in an embodiment of this application. The electronic rearview mirror includes at least the following structures:

[0167] The system comprises a controller 1301, at least one display 1302, and at least one image acquisition device 1303. The image acquisition device 1303 is connected to the controller 1301, and the controller 1301 is connected to the display 1302. The structure of the image acquisition device 1303 can be referenced. Figure 9 As shown, it includes at least: an image sensor;

[0168] The structure of the image sensor can be referred to Figure 2 As shown, it may include at least:

[0169] The sensor core module 201 is used to convert the acquired optical signals into digital electrical signals;

[0170] Image signal processing module 202 is connected to sensor core module 201 and is used to perform signal optimization processing on digital electrical signals.

[0171] The signal encoding module 203 is connected to the image signal processing module 202. The signal encoding module 203 is used to encode the optimized digital electrical signal to obtain image data.

[0172] Data interface 204 is connected to signal encoding module 203. Data interface 204 is used to transmit image data to controller 1301 connected to image acquisition device 1303.

[0173] The controller 1301 is used to process image data and transmit the obtained target image to the display 1302;

[0174] Display 1302 is used to output the target image.

[0175] By employing the above technical solution, in an electronic rearview mirror provided in this application embodiment, the signal encoding module is deployed within the image sensor included in the image acquisition device of the electronic rearview mirror. In this way, the signal encoding module and the image signal processing module within the image sensor are connected within the chip, enabling chip-level data transmission and encoding. Compared to deploying the signal encoding module on the printed circuit board where the image sensor is located to achieve on-board data transmission and encoding, chip-level data encoding and transmission can accelerate the image sensor's image data acquisition efficiency, thereby improving the image data output efficiency of the image acquisition device where the image sensor is located, and thus improving the image output efficiency of the electronic rearview mirror where the image acquisition device is located.

[0176] In one implementation, the controller 1301 can be a high-speed system-on-a-chip (SoC) chip with an optimized system architecture. Specifically, the controller 1301, such as a high-speed SoC chip, may include at least an in-memory computing accelerator 1311, such as... Figure 13 As shown:

[0177] The in-memory computing accelerator 1311 performs image processing on the image data corresponding to the image acquisition device 1303 to obtain an intermediate image, and stitches together the intermediate images corresponding to at least two image acquisition devices 1303 to obtain a target image.

[0178] As can be seen, in this embodiment, in-memory computing is performed in the controller through an in-memory computing accelerator to realize image processing and image stitching. By utilizing the extremely high energy efficiency, huge computing throughput, and low latency of the in-memory computing accelerator, the calculation of image processing and image stitching is integrated into the memory, and the image processing and image stitching are completed directly in the memory without the need to transfer data back and forth, thereby reducing the time consumption of data transfer. This can accelerate the implementation speed of image processing and image stitching in the controller, thereby improving the output efficiency of the controller.

[0179] In specific implementations, for example, such as Figure 14 As shown, a high-speed SoC chip may also include: a processor, a security management module, an on-chip module, a peripheral interface, a system bus, a codec, an external storage interface, a video input interface, and a video output interface.

[0180] The processor can be implemented through multiple ARM (Acorn RISC Machine) core chips, and the processor is used to implement system control and computing tasks of high-speed SoC chips.

[0181] The in-memory computing accelerator 1311 may include an image signal processor (ISP) and an image stitching module, which are used to perform image processing and image stitching computation tasks, respectively. Both the image signal processor and the image stitching module are capable of parallel computing, which effectively reduces the time required for algorithm computation and data transmission, and greatly improves the computing efficiency of the in-memory computing accelerator 1311.

[0182] The security management module is used to implement system security monitoring and processing tasks;

[0183] On-chip modules are used to implement routine system tasks and may include structures such as read-only memory (ROM), static random-access memory (SRAM), power management module, watchdog timer, clock management module, and temperature sensor.

[0184] The system bus is used to enable data / command communication between modules within the system;

[0185] The external storage interface is used to enable communication with external storage devices;

[0186] Codecs are used to encode and decode images and / or videos;

[0187] The video input interface is used to input image data (i.e., image data output by the image acquisition device);

[0188] The video output interface is used to output video data (i.e., the target image output by the in-memory computing accelerator);

[0189] Peripheral interfaces are used to connect to other chips that are different from high-speed SoC chips.

[0190] Based on the above implementation scheme, the in-memory computing accelerator 1311 includes at least one in-memory computing unit 1501, and the in-memory computing unit 1501 can consist of one or more units, such as... Figure 15 As shown, each in-memory computing unit 1501 can perform image processing computing tasks and image stitching computing tasks through multiple computing units and weight storage units;

[0191] The weight storage unit stores the image data, and multiple computing units update the feature data in the image data in the weight storage unit. Thus, the computing units in the in-memory computing unit 1501 directly execute computing tasks in the weight storage unit. During the execution of the computing task, each computing unit only needs to update the feature data in the image data, thereby eliminating the need to move the image data back and forth, reducing the time consumed by data movement, and enabling image processing and image stitching. This accelerates image processing and image stitching, thereby improving the efficiency of the controller in obtaining the target image.

[0192] It should be noted that the computing units in the in-memory computing unit 1501 can perform corresponding image processing and image stitching computing tasks through the deployed intelligent model.

[0193] Specifically, such as Figure 15 As shown, the in-memory computing accelerator 1311 may also include a system control unit 1502, an accumulator 1503, a pooler 1504, a multiply-accumulate unit 1505, a quantizer 1506, an activator 1507, an activation value buffer 1508, a high-speed data input interface 1509, and a high-speed data output interface 1510.

[0194] The system control unit 1502 is used to implement the system logic control of the in-memory computing accelerator 1311. The in-memory computing unit 1501 is the core module, which implements caching and computation on the ultra-high-speed memory. The accumulator 1503, pooler 1504, multiply-accumulate 1505, quantizer 1506, and activator 1507 are used to implement operations such as accumulation, pooling, multiply-accumulate, quantization, and activation, respectively.

[0195] The activation value cache 1508 is used to cache activation values. The activation value cache 1508 can use a dual-cache mode, such as cache 1 and cache 2, enabling full-duplex data transmission with the in-memory computing unit 1501. Figure 16 As shown, this greatly improves data transmission efficiency.

[0196] The high-speed data input interface 1509 and the high-speed data output interface 1510 are used to implement data input and output, respectively.

[0197] like Figure 16 As shown, the in-memory computing unit 1501 consists of computing units and weight storage units. The in-memory computing unit integrates multiple computing units that can perform various computational operations, such as matrix operations, vector operations, and logical operations, for performing complex computational tasks, such as image processing and image stitching. The weight storage unit within the in-memory computing unit stores image data and intermediate results required for the computational tasks.

[0198] The weighted storage unit supports various storage media, including volatile and non-volatile memory. Volatile memory includes Synchronous Dynamic Random-Access Memory (SDRAM) and Dynamic Random-Access Memory (DRAM). Non-volatile memory includes Resistive Random-Access Memory (RRAM), Magnetoresistive Random-Access Memory (MRAM), Ferroelectric Random-Access Memory (FRAM), and Phase-Change Random-Access Memory (PCRAM).

[0199] For example, such as Figure 17 As shown, traditional image processors such as ISPs integrate numerous algorithm processing modules, such as black level correction, lens correction, bad pixel correction, HDR, de-mosaic, noise reduction, white balance, color correction, gamma, color space conversion, and sharpening modules. These modules exchange a large amount of data, and each module performs matrix calculations independently, making the transmission and computation times incompressible. For example, ... Figure 18 As shown, a traditional image stitching module consists of a distortion correction module, a perspective transformation module, an image cropping module, an image fusion module, and an image scaling module. There is a large amount of data transmission between these modules. The image data input from the cameras deployed on the front, left, and right sides are processed in three pipelines, such as distortion correction, perspective transformation, cropping, color balancing, and weighted fusion based on intrinsic and extrinsic parameters. After scaling, the stitched image is finally output.

[0200] Compared to Figure 17 and Figure 18 In this embodiment Figure 16 The in-memory computing unit 1501 shown can save data transmission between computing units, thereby greatly improving the efficiency of image processing and image stitching.

[0201] Based on this, in this embodiment, when the SoC performs image processing and image stitching in the in-memory computing accelerator, it processes according to a 3×3 pixel matrix, and the corresponding image processing delay calculation is shown in formula (3):

[0202] T 处理 = 3 × T line (3)

[0203] Among them, T 处理 This refers to the SoC processing time, specifically the SoC processing time T2 mentioned earlier; T line The time taken to transmit one line of image data.

[0204] In one implementation, the controller 1301 may further include a signal decoding module 1312, which can be a decoder. The signal decoding module 1312 corresponds to the signal encoding module 203 in the image acquisition device 1303, such as... Figure 19 As shown, the signal decoding module 1312 is used to decode the image data output by the image acquisition device 1303 and transmit the decoded image data to the in-memory computing accelerator 1311. The in-memory computing accelerator 1311 performs image processing and image stitching tasks on the image data through the in-memory computing unit 1501 and other structures, thereby enabling the in-memory computing accelerator 1311 to output the target image.

[0205] As can be seen, the image data transmitted by the image acquisition device 1303 (e.g., a camera) is decoded by the signal decoding module 1312 (e.g., a decoder) of the controller 1301 and then directly transmitted to the in-memory computing accelerator 1311 (e.g., a SoC) within the controller 1301. The input image data does not need to pass through the internal or external memory cache of the chip, thereby reducing the data transmission time. In other words, in this embodiment, the decoded image data is directly transmitted to the in-memory computing accelerator through the signal decoding module, without going through the data cache. This reduces the read and write process of image data in the cache, thereby improving the image output efficiency of the controller by reducing the image data transmission time.

[0206] In one implementation, the controller 1301 may further include a serializer 1313, while the display 1302 may include a deserializer 1321 and a display panel 1322, such as an LCD panel. Figure 19 As shown, serializer 1313 is connected to deserializer 1321. Based on this, serializer 1313 converts the target image obtained by in-memory computing accelerator 1311 into serial data and transmits the serial data to deserializer 1321 in display 1302. Deserializer 1321 converts the serial data into parallel data and transmits the parallel data to display panel 1322. Display panel 1322 refreshes the screen based on the parallel data. Thus, after the image data is processed by in-memory computing accelerator, the target image is directly output to the display for display. The output target image does not need to pass through the internal or external memory cache of the chip, thereby reducing the data transmission time.

[0207] As can be seen, in this embodiment, the target image is directly transmitted between the controller and the display through a serializer and a deserializer, without going through the memory cache. This reduces the reading and writing process of the image in the cache, thereby reducing the image transmission time and improving the display's output efficiency.

[0208] Based on this, in this embodiment, the display 1302 does not need to cache after receiving the target image, and directly refreshes and displays it line by line, which greatly reduces the display time and thus improves the display efficiency. The display time can be calculated by formula (4):

[0209] T 显示 = 1 × T line (4)

[0210] Among them, T 显示 The time taken is T3 as mentioned earlier; T line The time taken to transmit one line of image data.

[0211] Furthermore, the controller 1301 may also include a power management module 1314, such as... Figure 19 As shown, the power management module 1314 provides power signals to the controller 1301, the image acquisition device 1303 (such as a camera), and the display 1302 through the signal decoding module 1312 (such as a decoder), the in-memory computing accelerator 1311 (such as a SoC), and the serializer 1313.

[0212] In one implementation, the signal encoding module 203 in the image sensor of the image acquisition device 1303 in the electronic rearview mirror can be a serializer module or an optical communication module.

[0213] In one implementation, the image acquisition device 1303 may further include: a parameter storage unit, such as... Figure 9 As shown in the image.

[0214] As can be seen, in this embodiment, the startup parameters are stored in the parameter storage unit connected to the image sensor within the image acquisition device. Therefore, when the image acquisition device is powered on, the controller connected to the image acquisition device does not need to set the configuration parameters of each module of the image sensor. This eliminates the need for the controller to participate in reading, writing, and setting the startup parameters. Instead, the control unit within the image sensor directly reads the startup parameters from the parameter storage unit deployed in the image acquisition device and sets the configuration parameters of the corresponding modules. This achieves chip-level data reading, writing, and parameter setting, thereby accelerating the configuration parameter setting of each module in the image sensor, thus speeding up the image sensor's startup process. This, in turn, accelerates the image sensor's light signal acquisition, signal conversion, signal optimization, and data encoding, thereby increasing the output efficiency of image data in the image sensor. Ultimately, this improves the image output efficiency of the image acquisition device and improves the image output efficiency of the electronic rearview mirror.

[0215] Furthermore, the parameter storage unit can also store system parameters and user parameters, such as... Figure 10 As shown in the image.

[0216] In one implementation, the image sensor in the image acquisition device 1303 of this embodiment is deployed as a pixel layer 1101, a logic circuit layer 1102, and a cache layer 1103. The structure of the pixel layer 1101, the logic circuit layer 1102, and the cache layer 1103 can be referred to Figure 11 .

[0217] The following provides a detailed example of the electronic rearview mirror according to an embodiment of this application:

[0218] like Figure 20 As shown, the electronic rearview mirror may include multiple cameras, a controller, and a display. The controller deploys a decoder (i.e., a signal decoding module), a SoC, a serializer, and a power management module. The display deploys a deserializer and an LCD panel (i.e., display panel 1322). Figure 21 As shown, the camera consists of an image sensor, a power supply unit, a clock unit, a storage unit (i.e., the parameter storage unit mentioned earlier), and a data interface. The image sensor performs three functions: first, it converts light signals into analog electrical signals, and then into digital electrical signals; second, it optimizes the processing of the digital electrical signals; and third, it encodes and outputs the digital electrical signals. The power supply unit provides operating power to all components. The clock unit provides the clock source signal for each module. The storage unit stores the image sensor's firmware parameters, such as startup parameters, system parameters, and user parameters. The data interface connects the camera to external wiring harnesses, such as a controller.

[0219] The camera and controller are connected via coaxial cable or fiber optic cable. The camera can transmit image data to the controller via the cable, and the controller can transmit control signals to the camera via the cable. The SoC chip in the controller uses in-memory computing technology to optimize the chip architecture, enabling rapid processing of functional modules such as data caching, image processing, image distortion correction, image scaling, and image stitching. This greatly saves data caching and transmission time, significantly improving data processing efficiency.

[0220] The controller and the display are connected via a coaxial cable. The controller transmits the processed target image to the display, which then displays the image. The controller outputs control signals to the display via a line. The display consists of a display controller, a deserializer, and an LCD panel. The display controller is used to control the display of the target image.

[0221] Specifically, the operation process of the electronic rearview mirror in this embodiment is divided into two stages: power-on startup and normal operation.

[0222] 1. Power-on startup phase:

[0223] System power-on: The power management module inside the controller powers on the camera, controller, and display.

[0224] Camera self-start: The control unit inside the camera reads the startup parameters from the parameter storage unit and completes the camera's initial configuration.

[0225] Controller startup and display startup: The control unit in the SoC configures the parameters of the SoC, other devices on the controller, and the display, and completes the initial configuration of the controller and display.

[0226] 2. Normal working phase:

[0227] After the camera is initialized, the light signal is converted into a digital electrical signal and encoded into image data, which is then transmitted to the SoC.

[0228] The SoC receives image data line by line and transmits it to the in-memory computing accelerator. After receiving three valid lines of image data, the image processor and image stitching module in the in-memory computing accelerator perform calculations and output the target image in real time.

[0229] The target image processed by the SoC is transmitted to the display line by line;

[0230] The monitor receives the target image line by line and displays it directly without buffering.

[0231] In summary, as Figure 22 The diagram shows the camera imaging time T1, SoC processing time T2, and display time T3 for an electronic rearview mirror that can achieve low-latency image transmission and display according to an embodiment of this application. As can be seen, this embodiment can effectively reduce the image output delay from image acquisition to screen display. The image output delay is calculated as shown in formula (1).

[0232] Taking a 3-megapixel resolution 60-frame image as an example, the camera imaging delay for one frame is 16.67ms, and the SoC processing delay for one line is 0.011ms. Therefore, the image output delay is as shown in formula (4):

[0233] T 系统 =16.67ms+3×0.011ms+1×0.011ms=16.714ms (4)

[0234] It is evident that the output delay of this application is no more than 20ms, which is significantly better than the output delay of other technical solutions that do not adopt the technical solution of this embodiment, such as 63.33ms.

[0235] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0236] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0237] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0238] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An image sensor, characterized in that, Deployed in an image acquisition device, the image acquisition device is equipped with a parameter storage unit. The image acquisition device is connected to a controller, and the controller is connected to a display. The parameter storage unit stores at least startup parameters, system parameters, and user parameters. The startup parameters are used to start the image acquisition device and are stored at the starting address of the parameter storage unit. The image sensor includes at least: The sensor core module is used to convert the acquired optical signals into digital electrical signals; An image signal processing module is used to perform signal optimization processing on the digital electrical signal; A signal encoding module is used to encode the optimized digital electrical signal to obtain image data; A data interface is used to transmit the image data to the controller, the controller processes the image data and transmits the resulting target image to the display, and the display outputs the target image. The control unit is used to respond to the start command of the image acquisition device, read the start parameters, the system parameters and the user parameters from the parameter storage unit, and set the configuration parameters corresponding to the sensor kernel module, the image signal processing module and the signal encoding module according to the start parameters, the system parameters and the user parameters.

2. The image sensor according to claim 1, characterized in that, The signal encoding module includes: a serializer module; the image data output by the serializer module is a digital differential signal; the serializer module includes: a data input interface, a control and data conversion module, and a serial interface; The data input interface is used to transmit the digital electrical signal from the image signal processing module to the control and data conversion module; The control and data conversion module is used to encode the digital electrical signal to obtain image data; The serial interface is used to transmit the image data to the data interface.

3. The image sensor according to claim 1, characterized in that, The signal encoding module includes: an optical communication module; the image data output by the optical communication module is an optical signal; the optical communication module includes: a data transmission interface, a data codec, an optical signal processor, an optical communication interface, and a control module; The data transmission interface is used to transmit the digital electrical signal from the image signal processing module to the data codec. The data codec is used to encode the digital electrical signal using an algorithm, and transmit the encoded image data to the optical signal processor. The optical signal processor is used to convert the image data encoded by the data codec into optical signals; The optical communication interface is used to transmit image data from the optical signal processor to the data interface.

4. The image sensor according to claim 1 or 2, characterized in that, The image sensor is deployed as a pixel layer, a logic circuit layer, and a cache layer; the sensor kernel module is deployed as a signal conversion module in the pixel layer and the logic circuit layer; the image signal processing module is deployed as an image signal processor in the logic circuit layer; wherein: The pixel layer is used to collect optical signals and convert the optical signals into analog charge signals; The signal conversion module of the logic circuit layer is used to convert the analog charge signal into a digital electrical signal; The image signal processor in the logic circuit layer is used to perform signal optimization processing on the digital electrical signal; The cache layer is used to store the digital electrical signals output by the logic circuit layer; the logic circuit layer reads the digital electrical signals from the cache layer and transmits them to the signal encoding module.

5. The image sensor according to claim 4, characterized in that, When the logic circuit layer converts the analog charge signal transmitted from the pixel layer into a digital electrical signal and transmits it to the signal encoding module, the image data obtained by the signal encoding module through data encoding of the digital electrical signal is the first frame rate. When the logic circuit layer reads the digital electrical signal from the cache layer and transmits it to the signal encoding module, the image data obtained by the signal encoding module through data encoding of the digital electrical signal is the second frame rate; The second frame rate is greater than the first frame rate.

6. An electronic rearview mirror, characterized in that, include: The system includes a controller, at least one display, and at least one image acquisition device, wherein the image acquisition device is connected to the controller, and the controller is connected to the display. The image acquisition device includes at least one of the following: an image sensor as described in any one of claims 1 to 5.

7. The electronic rearview mirror according to claim 6, characterized in that, The controller includes at least: an in-memory computing accelerator and a signal decoding module; The signal decoding module is used to decode the image data obtained by the image acquisition device through data encoding and transmit the decoded image data to the in-memory computing accelerator. The in-memory computing accelerator performs image processing on the image data transmitted by the signal decoding module to obtain an intermediate image, and stitches at least two of the intermediate images together to obtain the target image. The in-memory computing accelerator includes: an image processor and an image stitching module; The image processor is used to perform the image processing computation task; the image stitching module is used to perform the image stitching computation task; the image processor and the image stitching module perform computation in parallel.

8. The electronic rearview mirror according to claim 7, characterized in that, The in-memory computing accelerator includes at least an in-memory computing unit; the in-memory computing unit performs the image processing computing task and the image stitching computing task through multiple computing units and a weight storage unit; The weight storage unit is used to store the image data, and the plurality of computing units update the feature data in the image data in the weight storage unit.

9. The electronic rearview mirror according to claim 8, characterized in that, The multiple computing units perform the image processing and image stitching computation tasks through a deployed intelligent model.

10. The electronic rearview mirror according to claim 8, characterized in that, The in-memory computing accelerator also includes: Accumulator, pooler, multiply-accumulator, quantizer, activator, activation value buffer; The accumulator is used to accumulate the data output by the in-memory computing unit; the pooler is used to pool the data output by the accumulator; the multiply-accumulate is used to multiply-accumulate the data output by the pooler; the quantizer is used to quantize the data output by the multiply-accumulate; the activator is used to activate the data output by the quantizer; and the activation value buffer is used to store the activation value corresponding to the activator and provide the activation value to the in-memory computing unit.

11. The electronic rearview mirror according to claim 10, characterized in that, The activation value cache area transmits data in full-duplex with the in-memory computing unit through a dual-cache area mode.

12. The electronic rearview mirror according to any one of claims 7 to 11, characterized in that, The controller also includes a serializer; the display includes a deserializer and a display panel; the serializer is connected to the deserializer. The serializer converts the target image obtained by the in-memory computing accelerator into serial data and transmits the serial data to the deserializer. The deserializer converts the serial data into parallel data and transmits the parallel data to the display panel. The display panel refreshes the screen based on the parallel data.

Images