Data fusion method, device, electronic device and storage medium in sensor
By determining the proximity area and bias coefficient between chips and adjusting the brightness value, the brightness imbalance and splitting problems during multi-chip splicing in pulse sensors are solved, and data quality is improved.
Patent Information
- Application Number
- CN202311126475.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-01
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2043-09-01
AI Technical Summary
When multi-chip splicing in existing pulse sensors, due to systematic deviations between chips, brightness value imbalance and image fragmentation are caused, data quality is poor, and effective data fusion algorithms are lacking.
By determining the proximal area and bias coefficient between chips, adjusting the brightness value of the pixel area based on the brightness correlation value, fusion of multi-chip data is achieved, jumping at the chip junction is eliminated, and data quality is improved.
The pixel brightness uniformity of multi-chip output is achieved, which eliminates system errors and improves the quality and accuracy of sensor output data.
Smart Images

Figure CN117135477B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to sensor technology, and in particular to a data fusion method, device, electronic device and storage medium in a sensor. Background Art
[0002] Pulse sensors are considered to be one of the strong candidates for the new generation of visual sensors, which are closer to human eyes in principle. Pulse sensors use pulse streams to approximate the high-speed spatiotemporal changes of light, and can directly collect photon streams in the real world, thereby retaining complete visual information, which has significant advantages, especially for high-speed motion scenes. Summary of the invention
[0003] Embodiments of the present disclosure provide a data fusion method, device, electronic device, and storage medium in a sensor.
[0004] According to one aspect of an embodiment of the present disclosure, a data fusion method in a sensor is provided, including:
[0005] Based on the positional relationship between the plurality of chips distributed in the array in the sensor, at least one adjacent region corresponding to each chip is determined to obtain a plurality of adjacent regions; wherein the adjacent regions are located adjacent to the pixel regions corresponding to two of the chips;
[0006] In response to the pre-stored database not including the bias coefficient between every two adjacent chips among the plurality of chips, determining the bias coefficient between every two adjacent chips among the plurality of chips based on the brightness correlation values in the plurality of adjacent areas; wherein each of the bias coefficients represents a performance error of one chip relative to the other chip among the two adjacent chips;
[0007] Based on the plurality of bias coefficients, adjusting the brightness value of at least one pixel area among the plurality of pixel areas;
[0008] The data output by the multiple chips are fused based on the adjusted pixel area.
[0009] Optionally, the determining at least one adjacent region corresponding to each chip based on the positional relationship between the plurality of chips distributed in an array in the sensor to obtain the plurality of adjacent regions includes:
[0010] Based on the positional relationship between the plurality of chips distributed in an array in the sensor, determining at least one adjacent chip corresponding to each chip in the plurality of chips;
[0011] Based on a pixel area of a preset size that is adjacent to a pixel area corresponding to a chip located adjacent to the chip in the pixel area corresponding to each chip, the at least one adjacent area corresponding to the chip is determined to obtain a plurality of adjacent areas.
[0012] Optionally, the brightness-related value is a brightness value;
[0013] The step of determining the bias coefficient between every two adjacent chips in the plurality of chips based on the brightness correlation values in the plurality of adjacent areas comprises:
[0014] Based on the brightness values in the plurality of adjacent areas, determining a brightness and value corresponding to each of the adjacent areas;
[0015] Based on the brightness and value of two adjacent adjacent areas corresponding to the two adjacent chips, the bias coefficients corresponding to the two adjacent chips are determined.
[0016] Optionally, determining the brightness and value corresponding to each adjacent area based on the brightness values in the plurality of adjacent areas includes:
[0017] Sum the brightness values corresponding to all pixels included in the adjacent area to obtain the brightness sum value corresponding to the adjacent area; or,
[0018] The brightness values corresponding to all pixels included in the adjacent area are weighted to obtain the brightness sum value corresponding to the adjacent area.
[0019] Optionally, the brightness related value is pulse data;
[0020] The step of determining the bias coefficient between every two adjacent chips in the plurality of chips based on the brightness correlation values in the plurality of adjacent areas comprises:
[0021] Based on the pulse data in the plurality of adjacent regions, determining a pulse sum value corresponding to each of the adjacent regions;
[0022] Based on the pulse sum values of two adjacent adjacent areas corresponding to the two adjacent chips, the bias coefficients corresponding to the two adjacent chips are determined.
[0023] Optionally, adjusting the brightness value of at least one pixel area among the multiple pixel areas based on the multiple bias coefficients includes:
[0024] Determine one chip from the plurality of chips as a first chip and the other chips as second chips;
[0025] Determine a correction value corresponding to each second chip based on a plurality of bias coefficients corresponding to a plurality of second chips;
[0026] The brightness value in the pixel area corresponding to the second chip is adjusted based on the correction value to obtain at least one adjusted pixel area.
[0027] Optionally, determining the correction value corresponding to each second chip based on the plurality of bias coefficients corresponding to the plurality of second chips includes:
[0028] In response to the second chip being adjacent to the first chip, determining a correction value corresponding to the second chip based on a bias coefficient corresponding to the second chip;
[0029] In response to the second chip not being adjacent to the first chip, a correction value corresponding to the second chip is determined based on at least one path bias coefficient corresponding to the second chip.
[0030] Optionally, before determining the correction value corresponding to the second chip based on the at least one path bias coefficient corresponding to the second chip, the method further includes:
[0031] Determine at least one path indirectly connected from the second chip to the first chip; wherein each of the paths includes at least one intermediate chip;
[0032] The path bias coefficient is determined based on the bias coefficient corresponding to the second chip and the bias coefficient corresponding to each intermediate chip included in each of the paths; wherein each of the paths corresponds to one path bias coefficient.
[0033] Optionally, determining the correction value corresponding to the second chip based on at least one path bias coefficient corresponding to the second chip includes:
[0034] determining a correction value corresponding to the second chip based on one of the at least one path bias coefficient; or,
[0035] A geometric mean is calculated for the at least one path bias coefficient, and a correction value corresponding to the second chip is determined using a value obtained by the geometric mean.
[0036] Optionally, the brightness-related value is a brightness value;
[0037] The step of determining the bias coefficient between every two adjacent chips in the plurality of chips based on the brightness correlation values in the plurality of adjacent areas comprises:
[0038] Constructing a linear model for the brightness values in two adjacent adjacent regions by linear fitting;
[0039] The bias coefficient is determined based on the parameters in the linear model.
[0040] Optionally, constructing a linear model for the brightness values in two adjacent adjacent regions by linear fitting includes:
[0041] Determine the linear model of the preset structure;
[0042] Determining parameters in the linear model based on all brightness values in two adjacent adjacent regions;
[0043] Substitute the parameters into the linear model to obtain the linear model with known parameters.
[0044] Optionally, adjusting the brightness value of at least one pixel area among the multiple pixel areas based on the multiple bias coefficients includes:
[0045] Determine one chip from the plurality of chips as a first chip and the other chips as second chips;
[0046] Based on the linear models corresponding to every two adjacent chips, determining an updated linear model from each second chip to the first chip;
[0047] The brightness values in the pixel areas corresponding to each of the second chips are respectively brought into the corresponding updated linear model to obtain a plurality of adjusted brightness values corresponding to the pixel areas.
[0048] Optionally, it also includes:
[0049] In response to the pre-stored database including a bias coefficient between every two adjacent chips among the plurality of chips;
[0050] The brightness value of at least one adjacent area among the plurality of adjacent areas is adjusted based on the plurality of pre-stored bias coefficients.
[0051] Optionally, it also includes:
[0052] The bias coefficient between every two adjacent chips in the plurality of chips is stored in the pre-stored database.
[0053] According to another aspect of an embodiment of the present disclosure, a data fusion device in a sensor is provided, comprising:
[0054] A region determination module, used to determine at least one adjacent region corresponding to each chip based on the positional relationship between the plurality of chips distributed in an array in the sensor, so as to obtain a plurality of adjacent regions; wherein the adjacent region is located adjacent to the pixel regions corresponding to two of the chips;
[0055] A bias coefficient determination module, configured to determine the bias coefficient between every two adjacent chips among the plurality of chips based on the brightness correlation values in the plurality of adjacent areas in response to the pre-stored database not including the bias coefficient between every two adjacent chips among the plurality of chips; wherein each of the bias coefficients represents a performance error of one chip relative to the other chip among the two adjacent chips;
[0056] a brightness adjustment module, configured to adjust a brightness value of at least one pixel area among the plurality of pixel areas based on the plurality of bias coefficients;
[0057] The data fusion module is used to fuse the data output by the multiple chips based on the adjusted pixel area.
[0058] Optionally, the area determination module is specifically used to determine at least one adjacent chip corresponding to each of the multiple chips based on the positional relationship between the multiple chips distributed in an array in the sensor; and to determine at least one adjacent area corresponding to the chip based on a pixel area of a preset size in a pixel area corresponding to each chip that is connected to a pixel area corresponding to the adjacent chip, so as to obtain a plurality of adjacent areas.
[0059] Optionally, the brightness-related value is a brightness value;
[0060] The bias coefficient determination module comprises:
[0061] a brightness sum value determining unit, configured to determine the brightness sum value corresponding to each of the adjacent areas based on the brightness values in the plurality of adjacent areas;
[0062] The first bias coefficient unit is used to determine the bias coefficients corresponding to the two adjacent chips based on the brightness and value of the two adjacent adjacent areas corresponding to the two adjacent chips.
[0063] Optionally, the sum value determination unit is specifically used to sum the brightness values corresponding to all pixels included in the adjacent area to obtain the brightness sum value corresponding to the adjacent area; or, to weight the brightness values corresponding to all pixels included in the adjacent area to obtain the brightness sum value corresponding to the adjacent area.
[0064] Optionally, the brightness related value is pulse data;
[0065] The bias coefficient determination module comprises:
[0066] A pulse sum value determination unit, configured to determine a pulse sum value corresponding to each adjacent area based on the pulse data in the plurality of adjacent areas;
[0067] The second bias coefficient unit is used to determine the bias coefficients corresponding to the two adjacent chips based on the pulse sum values of the two adjacent adjacent areas corresponding to the two adjacent chips.
[0068] Optionally, the brightness adjustment module includes:
[0069] A reference determination unit, configured to determine one chip from the plurality of chips as a first chip and the other chips as second chips;
[0070] A correction value unit, configured to determine a correction value corresponding to each of the second chips based on a plurality of bias coefficients corresponding to the plurality of second chips;
[0071] The area adjustment unit is used to adjust the brightness value in the pixel area corresponding to the second chip based on the correction value to obtain at least one adjusted pixel area.
[0072] Optionally, the correction value unit is specifically used to determine the correction value corresponding to the second chip based on the bias coefficient corresponding to the second chip in response to the second chip being adjacent to the first chip; and to determine the correction value corresponding to the second chip based on at least one path bias coefficient corresponding to the second chip in response to the second chip being not adjacent to the first chip.
[0073] Optionally, the brightness adjustment module further includes:
[0074] A path bias determination unit is used to determine at least one path indirectly connected from the second chip to the first chip; wherein each of the paths includes at least one intermediate chip; based on the bias coefficient corresponding to the second chip and the bias coefficient corresponding to each intermediate chip included in each of the paths, determine the path bias coefficient; wherein each of the paths corresponds to one path bias coefficient.
[0075] Optionally, when the correction value unit determines the correction value corresponding to the second chip based on at least one path bias coefficient corresponding to the second chip, the correction value unit is used to determine the correction value corresponding to the second chip based on one of the at least one path bias coefficient; or, to obtain the geometric mean of the at least one path bias coefficient and determine the correction value corresponding to the second chip with the value obtained by the geometric mean.
[0076] Optionally, the brightness-related value is a brightness value;
[0077] The bias coefficient determination module comprises:
[0078] A linear model unit, used for constructing a linear model for the brightness values in two adjacent adjacent areas through linear fitting;
[0079] The third bias coefficient unit is used to determine the bias coefficient based on the parameters in the linear model.
[0080] Optionally, the linear model unit is specifically used to determine the linear model of the preset structure; determine the parameters in the linear model based on all brightness values in two adjacent adjacent areas; and bring the parameters into the linear model to obtain the linear model with known parameters.
[0081] Optionally, the brightness adjustment module is specifically used to determine one chip from the multiple chips as the first chip, and the other chips as the second chip; based on the linear model corresponding to every two adjacent chips, determine the updated linear model of each second chip to the first chip; respectively bring the brightness value in the pixel area corresponding to each second chip into the corresponding updated linear model to obtain multiple adjusted brightness values corresponding to the pixel area.
[0082] Optionally, the device further comprises:
[0083] A pre-stored coefficient module, used for responding to the bias coefficient between every two adjacent chips in the plurality of chips included in the pre-stored database;
[0084] The brightness adjustment module is used to adjust the brightness value of at least one adjacent area among the multiple adjacent areas based on the multiple pre-stored bias coefficients.
[0085] Optionally, the device further comprises:
[0086] The coefficient storage module is used to store the bias coefficient between every two adjacent chips in the plurality of chips into the pre-stored database.
[0087] According to another aspect of the embodiments of the present disclosure, there is provided an electronic device, comprising: a processor, and a memory communicatively connected to the processor, and further comprising the data fusion method in a sensor as described in any one of the above embodiments;
[0088] The memory stores computer-executable instructions;
[0089] The processor executes computer-executable instructions stored in the memory to control a data fusion method in the sensor.
[0090] Optionally, the electronic device is included in any one of the following: a pulse camera, a high-speed camera, an audio / video player, a navigation device, a fixed-position terminal, an entertainment unit, a smart phone, a communication device, a device in a motor vehicle, a camera, a sports or wearable camera, a detection device, a flight device, a medical device, a security device.
[0091] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the instructions are executed, the computer executes the data fusion method in the sensor described in any one of the embodiments.
[0092] Based on the data fusion method, device, electronic device and storage medium in the sensor provided by the above embodiments of the present disclosure, based on the positional relationship between the multiple chips distributed in the array in the sensor, at least one adjacent area corresponding to each chip is determined to obtain multiple adjacent areas; wherein the adjacent area is located adjacent to the pixel areas corresponding to the two chips; in response to the bias coefficient between each two adjacent chips in the multiple chips not included in the pre-stored database, based on the brightness correlation value in the multiple adjacent areas, the bias coefficient between each two adjacent chips in the multiple chips is determined; wherein each bias coefficient represents the error of one chip relative to the other chip in the two adjacent chips; based on the multiple bias coefficients, the brightness value of at least one pixel area in the multiple pixel areas is adjusted; based on the adjusted pixel area, the data output by the multiple chips is fused. The embodiment of the present disclosure adjusts the brightness value in the adjacent area by the bias coefficient, so that the brightness values in the pixel areas corresponding to the multiple chips in the sensor are unified, solves the problem of brightness value jump at the chip junction, makes the pixel brightness transition of the multi-chip output natural, and improves the quality of the sensor output data.
[0093] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0094] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0095] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:
[0096] Figure 1-1 It is a schematic diagram of directly fusing the data output by multiple sensor chips based on the existing technology;
[0097] Figure 1-2 It is another schematic diagram of directly fusing the data output by multiple sensor chips based on the prior art;
[0098] Figure 2 is a flow chart of a data fusion method in a sensor provided by an exemplary embodiment of the present disclosure;
[0099] Figure 3is a schematic diagram of an optional multi-chip array in a data fusion method in a sensor provided by an exemplary embodiment of the present disclosure;
[0100] Figure 4-1 is an optional schematic diagram of implementing data fusion based on the data fusion method in a sensor provided by an exemplary embodiment of the present disclosure;
[0101] Figure 4-2 is another optional schematic diagram of implementing data fusion based on the data fusion method in a sensor provided by an exemplary embodiment of the present disclosure;
[0102] Figure 5-1 is a flow chart of step 202 in the embodiment shown in FIG. 1 of the present disclosure;
[0103] Figure 5-2 Yes Figure 3 An optional schematic diagram of multiple chips defining adjacent regions shown;
[0104] Figure 6-1 is a flowchart of step 204 in the embodiment shown in FIG. 1 of the present disclosure;
[0105] Figure 6-2 is another flowchart diagram of step 204 in the embodiment shown in FIG. 1 of the present disclosure;
[0106] Figure 7 is a flow chart of step 206 in the embodiment shown in FIG. 1 of the present disclosure;
[0107] Figure 8 is another flowchart diagram of step 204 in the embodiment shown in FIG. 1 of the present disclosure;
[0108] Fig. 9 is a structural schematic diagram of a data fusion device in a sensor provided by an exemplary embodiment of the present disclosure;
[0109] Fig.10 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated. DETAILED DESCRIPTION
[0110] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.
[0111] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
[0112] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.
[0113] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.
[0114] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.
[0115] In addition, the term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the previous and next associated objects are in an "or" relationship. The data referred to in this disclosure may include unstructured data such as text, images, and videos, and may also be structured data.
[0116] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.
[0117] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0118] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.
[0119] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0120] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0121] In the process of realizing the present disclosure, the inventors found that in the field of multi-chip sensors such as existing pulse camera sensors, due to current process and cost limitations, it is necessary to use a method of splicing multiple chips to realize sensor functions. However, in actual applications, due to errors or other reasons when each chip is taped out, the data collected by each chip has systematic deviations. For example, in a pulse sensor, the perception threshold of a certain chip is low, resulting in the area corresponding to this chip emitting more pulses than other areas under the same light intensity, which in turn causes the brightness of the reconstructed image to be unbalanced, and the areas are clearly separated. For example, Figure 1-1 As shown in the example provided in 1-2, the image obtained by directly fusing the data collected by the four chips has obvious separation at the junction of the chips. If the data sensed by these chips is directly used as the sensor output, the data will jump significantly at the junction of the chips, and the data quality will be poor, with systematic deviations.
[0122] Since pulse sensor is a brand-new technology, there is no relevant algorithm for multi-chip pulse sensor data fusion at present. The present disclosure proposes for the first time a method for solving the deviation between chips of multi-chip pulse sensor.
[0123] Figure 2 FIG. 1 is a flow chart of a data fusion method in a sensor provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 2 As shown, the following steps are included:
[0124] Step 202: Based on the positional relationship between the plurality of chips distributed in the array in the sensor, at least one adjacent region corresponding to each chip is determined to obtain a plurality of adjacent regions.
[0125] The adjacent area is located adjacent to the pixel areas corresponding to the two chips.
[0126] In this embodiment, each chip in the plurality of chips distributed in the array is adjacent to at least one chip, for example, Figure 3 As shown, the sensor includes 4 chips, namely chip ①, ②, ③, and ④; it can be seen from the figure that chip ① is adjacent to chip ② and chip ④, chip ② is adjacent to chip ① and chip ③, chip ③ is adjacent to chip ② and chip ④, and chip ④ is adjacent to chip ① and chip ③; a part of pixels of a preset size can be determined at the position where each chip is adjacent to other chips as an adjacent area, that is, chips ①, ②, ③, and ④ correspond to two adjacent areas, respectively.
[0127] Step 204 , in response to the pre-stored database not including the bias coefficient between every two adjacent chips in the plurality of chips, determine the bias coefficient between every two adjacent chips in the plurality of chips based on the brightness correlation values in the plurality of adjacent areas.
[0128] Among them, each bias coefficient represents the brightness performance error of one chip relative to the other chip in two adjacent chips; the brightness performance error is the performance error caused when each chip is taped out. This embodiment mainly focuses on the brightness performance error in the performance error. For example, the brightness performance error can reflect the difference between the brightness values corresponding to the data collected by the two chips; optionally, the brightness-related value can be brightness value, pulse data, and other data related to image brightness.
[0129] The bias coefficient in this embodiment supports real-time acquisition or non-real-time acquisition. Non-real-time acquisition means that before the data fusion is performed this time, the bias coefficients between the multiple chips corresponding to the sensor are determined by other data and stored in the pre-stored database. When the data fusion is performed this time, the bias coefficient corresponding to each chip is directly obtained from the pre-stored database to realize data fusion. The bias coefficient can be obtained based on at least one of the following methods:
[0130] Bias calculation based on data set: By collecting data collected by the chip in different scenarios, the bias coefficient is calculated, and this bias coefficient is used to achieve data fusion when the sensor is subsequently applied. This is a non-real-time determination of the bias coefficient.
[0131] Bias calculation based on the current scene: After collecting data for a period of time in the current scene, the bias coefficient is calculated based on the data in this time period, and then the bias coefficient is applied to fuse the data collected by the sensor.
[0132] Bias calculation based on the current moment: Use the data at the current moment to calculate the bias coefficient in real time, and fuse the data at the current moment based on the bias coefficient calculated at the current moment to achieve real-time correction.
[0133] Step 206: Adjust the brightness value of at least one pixel area among the multiple pixel areas based on the multiple offset coefficients.
[0134] By analyzing and calculating the perception data at the junction of adjacent chips, the offset between each chip is derived. Based on the study of pulse sensor data, the main deviation between chips lies in the trigger threshold of the pulse sensor. After reconstructing the brightness matrix using the pulse interval texture (TFI, Texture from Interval) algorithm, the trigger threshold deviation of the pulse sensor can be expressed as the ratio of the brightness values of two pixels with the same true brightness in the brightness matrix (for example, the offset coefficient). Using the principle of spatial continuity to perform statistical analysis on the chip junction, the deviation between different chips of the pulse sensor can be well estimated. Therefore, adjusting the brightness value in the pixel area based on the offset coefficient can more accurately compensate for the systematic error between chips, thereby improving the quality of the sensor output data. By accurately identifying and compensating for the systematic error between chips, the quality and accuracy of the sensor output data can be improved.
[0135] Step 208: fusing the data output by the multiple chips based on the adjusted pixel area.
[0136] In some optional examples, after adjusting at least one pixel area, the brightness values of the pixel areas corresponding to all chips are unified. At this time, the data output by multiple chips are fused to solve the problem of separation at the junction of different chips. For example, by processing the embodiment of the present disclosure, Figure 1-1 The image in the embodiment shown can be obtained as follows Figure 4-1 The fused image shown is processed by the embodiment of the present disclosure. Figure 1-2 The image in the embodiment shown can be obtained as follows Figure 4-2 The fused image shown is Figure 4-1 and 4-2 It can be seen that after adjusting the brightness value of the pixel area by the method provided in this embodiment, an image with uniform brightness can be obtained by fusing the data output by multiple chips. There is no jump in the image, which effectively eliminates the system error between multi-chip sensors, improves the quality and accuracy of the sensor output data, and ensures real-time performance and computing efficiency.
[0137] The data fusion method in the sensor provided by the above-mentioned embodiment of the present disclosure determines at least one adjacent area corresponding to each chip based on the positional relationship between multiple chips distributed in an array in the sensor, and obtains multiple adjacent areas; wherein the adjacent area is located adjacent to the pixel areas corresponding to two chips; in response to the bias coefficient between each two adjacent chips in the multiple chips not included in the pre-stored database, the bias coefficient between each two adjacent chips in the multiple chips is determined based on the brightness correlation value in the multiple adjacent areas; wherein each bias coefficient represents the brightness performance error of one chip relative to the other chip in the two adjacent chips; based on the multiple bias coefficients, the brightness value of at least one adjacent area in the multiple adjacent areas is adjusted; based on the adjusted adjacent areas, the data output by the multiple chips are fused; the embodiment of the present disclosure adjusts the brightness value in the adjacent area by the bias coefficient, so that the brightness values in the pixel areas corresponding to the multiple chips in the sensor are unified, which solves the problem of brightness value jump at the chip junction, makes the pixel brightness transition of multiple chips output natural, and improves the quality of sensor output data.
[0138] like Figure 5-1 As shown in the above Figure 2 Based on the illustrated embodiment, step 202 may include the following steps:
[0139] Step 2021: Determine at least one adjacent chip corresponding to each chip in the plurality of chips based on the positional relationship between the plurality of chips distributed in an array in the sensor.
[0140] Optionally, the positional relationship between the multiple chips in the sensor determines the connection relationship between each chip and other chips, for example, Figure 3 The four chips shown can be determined to be adjacent to two chips based on the positional relationship between the four chips, specifically: chip ① is adjacent to chip ② and chip ④, chip ② is adjacent to chip ① and chip ③, chip ③ is adjacent to chip ② and chip ④, and chip ④ is adjacent to chip ① and chip ③.
[0141] Step 2022, based on a pixel area of a preset size in the pixel area corresponding to each chip that is adjacent to the pixel area corresponding to the adjacent chip, determine at least one adjacent area corresponding to the chip to obtain a plurality of adjacent areas.
[0142] In this embodiment, the area of a preset size at the position where each chip is adjacent to the adjacent chip is taken as the adjacent area. For example, chip ① is adjacent to chip ② and chip ④, then the pixel area corresponding to chip ① has two adjacent areas. For example, Figure 5-2 As shown, two adjacent regions (such as Figure 5-2The gray area in the chip ①, the adjacent area displayed vertically in chip ① represents the adjacent area from chip ① to chip ②, and the adjacent area displayed horizontally in chip ① represents the adjacent area from chip ① to chip ④; in addition, the size of the adjacent area can be preset according to the application scenario or set according to user needs. Since the length of the junction between any two chips is fixed, the preset size in this embodiment can be achieved by setting the preset width. For example, when the preset width is set to 1 pixel and the pixel area corresponding to each chip is 500 pixels x 500 pixels, the size of the adjacent area is 500 pixels x 1 pixel; of course, the preset width can be set to other values, such as 2, 3, 4, etc. The larger the value, the larger the area of the adjacent area obtained, which can improve the accuracy of the bias coefficient, but it will increase the amount of calculation. Therefore, the preset width can be set according to the accuracy requirements of the application scenario.
[0143] like Figure 6-1 As shown in the above Figure 2 Based on the embodiment shown, in an optional embodiment, the brightness-related value is a brightness value; step 204 may include the following steps:
[0144] Step 2041, based on the brightness values in multiple adjacent areas, determine the brightness and value corresponding to each adjacent area.
[0145] Optionally, the determination of brightness and value can be obtained by:
[0146] The brightness values corresponding to all pixels included in the adjacent area are summed to obtain the brightness sum value corresponding to the adjacent area.
[0147] In this embodiment, all brightness values are accumulated and summed, and the sum is used to represent the brightness sum corresponding to the adjacent area. In addition, a time dimension can be added to the summing operation to accumulate and sum the brightness values in the adjacent areas of any time length. In this embodiment, the brightness of the adjacent areas is represented by accumulating and summing the brightness values, thereby evaluating the brightness of the adjacent areas as a whole and improving the integrity of the brightness adjustment.
[0148] Alternatively, the brightness values corresponding to all pixels included in the adjacent area are weighted to obtain the brightness sum value corresponding to the adjacent area.
[0149] Optionally, the weighted processing may include but is not limited to weighted summation, weighted averaging, etc. Optionally, a weight value may be assigned to each pixel based on factors such as the signal-to-noise ratio of the data at the junction of adjacent chips and the perception range, and then the brightness values in the adjacent areas are weighted to determine the brightness and value corresponding to the adjacent areas.
[0150] Step 2042: Determine the bias coefficients corresponding to the two adjacent chips based on the brightness and value of the two adjacent adjacent areas corresponding to the two adjacent chips.
[0151] Optionally, the bias coefficients corresponding to two adjacent chips can be determined based on the ratio of the brightness and value of two adjacent adjacent areas. For example, the brightness and value corresponding to the adjacent area at the junction of chip 1 and chip 2 is SUM 12 The brightness and value of the adjacent area at the junction of chip 2 and chip 1 are SUM 21 , then the bias coefficient between chip 1 and chip 2 is SUM 12 / SUM 21 , and the bias coefficient between chip 2 and chip 1 is SUM 21 / SUM 12 Each bias coefficient of each chip relative to other adjacent chips can be determined based on the ratio calculation; when the time length is accumulated in the brightness and value, the bias coefficient is determined directly based on the ratio of the brightness and value accumulated for the same time length.
[0152] In an optional example, a 4-chip camera type sensor fusion is used as an example. The pixel of the camera is 1000 pixels x 1000 pixels, which is composed of 4 chips spliced together. The 4 chips are arranged in a 2x2 matrix (the distribution of the 4 chips can be referred to Figure 3 As shown in the figure), the pixel area corresponding to each chip is 500 pixels x 500 pixels.
[0153] First, each chip collects information from the real world and converts it into a 1000 pixel x 1000 pixel brightness value matrix. Then, the brightness and value of the adjacent pixels of any two adjacent chips are calculated respectively, and the ratio is calculated to obtain the bias coefficient. Optionally, in this embodiment, the preset size of the adjacent area is 500×1, that is, the preset width is 1 pixel; the determination of the brightness and value can be based on: Where B is the brightness value function, the subscripts 1, 2, etc. represent the chip number, Sum 21 Indicates the brightness and value of the adjacent area between chip ② and chip ①, Sum 12 represents the brightness and value of the adjacent area between chip ① and chip ②, i represents the pixel position in the adjacent area, and the value of i ranges from 1 to 500; (i, 500) represents the pixel in the 500th column in the pixel array of 1000 pixels x 1000 pixels, that is, Sum 21 What is calculated is the sum of the brightness values of the rightmost column of pixels in chip ②, which represents the sum of the brightness values of the adjacent area corresponding to chip ②. In this embodiment, the chips are numbered in the order of the rectangular coordinate system. Similarly, we can get They respectively represent the brightness and value of the adjacent area in chip ③ that is adjacent to chip ④, the brightness and value of the adjacent area in chip ④ that is adjacent to chip ③, the brightness and value of the adjacent area in chip ② that is adjacent to chip ③, the brightness and value of the adjacent area in chip ③ that is adjacent to chip ②, the brightness and value of the adjacent area in chip ① that is adjacent to chip ④, and the brightness and value of the adjacent area in chip ④ that is adjacent to chip ①.
[0154] Based on the above brightness and value, the bias coefficient between the two chips is calculated. They respectively represent the bias coefficient of chip ② relative to chip ①, the bias coefficient of chip ① relative to chip ②, and so on.
[0155] The bias coefficient characterizes the brightness difference between the two chips, and the bias coefficient is subsequently used to determine the compensation value corresponding to the data collected by each chip. This embodiment has high computational efficiency and low computational complexity, and performs well in computational efficiency and real-time performance, and is suitable for application scenarios with high real-time requirements.
[0156] like Figure 6-2 As shown in the above Figure 2 Based on the embodiment shown, in another optional embodiment, the brightness related value is pulse data; step 204 may include the following steps:
[0157] Step 2043, based on the pulse data in multiple adjacent areas, determine the pulse sum value corresponding to each adjacent area.
[0158] Step 2044, determining the bias coefficients corresponding to the two adjacent chips based on the pulse sum values of the two adjacent adjacent regions corresponding to the two adjacent chips.
[0159] In this embodiment, since the pixel area output by the pulse sensor is obtained by processing the pulse data output by the chip, this embodiment directly counts the number of pulses emitted in the pulse domain to determine the pulse sum value corresponding to each adjacent area. The ratio of the sum of the pulse emission data at the junction of adjacent chips can also be used to determine the bias coefficient between chips.
[0160] like Figure 7 As shown in the above Figure 2 Based on the illustrated embodiment, step 206 may include the following steps:
[0161] Step 2061: determine one chip from a plurality of chips as a first chip, and the other adjacent areas as second chips.
[0162] In this embodiment, it is necessary to select one chip from multiple chips as a benchmark, and unify the brightness values of other chips with the brightness of the benchmark chip. In this embodiment, a first chip is determined as the benchmark chip. The first chip can be randomly selected or determined based on any achievable regular method. After the first chip is determined, all other chips are second chips.
[0163] Step 2062: Determine a correction value corresponding to each second chip based on a plurality of bias coefficients corresponding to the plurality of second chips.
[0164] Step 2063: Adjust the brightness value in the pixel area corresponding to the second chip based on the correction value to obtain at least one adjusted pixel area.
[0165] In this embodiment, the correction value corresponding to each second chip is determined based on the bias coefficient, and the brightness value of each pixel in the corresponding pixel area is adjusted based on the correction value. For example, the brightness value of each pixel in the pixel area is multiplied by the corresponding correction value to achieve brightness adjustment of the pixel area; optionally, when the brightness value adjusted based on the correction value exceeds the maximum range of the brightness value, the adjusted brightness value is truncated. For example, when the maximum brightness value is 255, if the value obtained by multiplying the brightness value by the correction value is greater than 255, it can be truncated to 255 to avoid the problem of incorrect display; the brightness of the pixel area corresponding to each chip obtained by the processing of this embodiment is consistent with the brightness of the reference chip, so that the image quality output by the sensor is better.
[0166] Optionally, step 2062 may include:
[0167] In response to the second chip being adjacent to the first chip, determining a correction value corresponding to the second chip based on a bias coefficient corresponding to the second chip;
[0168] In response to the second chip not being adjacent to the first chip, a correction value corresponding to the second chip is determined based on at least one path bias coefficient corresponding to the second chip.
[0169] In this embodiment, since the first chip is determined, that is, the reference chip is determined, according to the distribution of the chips, some chips are directly connected to (adjacent to) the reference chip, for example, Figure 3 In the embodiment shown, chip ① is used as the reference chip, and chips ② and ④ are adjacent to the reference chip. The brightness values of the pixels in these chips are corrected directly based on the correction values of the chips relative to the reference chip. However, some chips are not directly connected to the reference chip (not adjacent), for example, Figure 3In the embodiment shown, chip ① is used as a reference chip, and chip ③ is not adjacent to the reference chip. For the non-adjacent chips, the correction values corresponding to these second chips can be determined by the path bias coefficient. Optionally, before determining the correction value corresponding to the second chip based on at least one path bias coefficient corresponding to the second chip, the method may further include:
[0170] At least one path indirectly connected from the second chip to the first chip is determined.
[0171] Each path includes at least one intermediate chip. Optionally, each non-adjacent second chip and the first chip form a path through at least one intermediate chip. For example, in the above embodiment, chip ③ is not adjacent to the reference chip, and two paths from chip ③ to chip ① can be determined. One path is through chip ② to chip ①, and chip ② is the intermediate chip of the path; the other path is through chip ④ to chip ①, and chip ④ is the intermediate chip of the path.
[0172] Based on the bias coefficient corresponding to the second chip and the bias coefficient corresponding to each intermediate chip included in each path, a path bias coefficient is determined; wherein each path corresponds to a path bias coefficient.
[0173] Optionally, a path determines all the intermediate chips passing from the second chip to the first chip, as well as the connection relationship between the second chip and the intermediate chip, the connection relationship between the intermediate chips (when more than two intermediate chips are included), and the connection relationship between the intermediate chip and the first chip. Based on these connection relationships, the bias coefficient of the second chip relative to the intermediate chip adjacent to it can be obtained, and the bias coefficient of the intermediate chip corresponding to the path direction, for example, includes two adjacent intermediate chips, one intermediate chip is adjacent to the second chip, and the other intermediate chip is adjacent to the first chip. The path bias coefficient can be determined based on a first bias coefficient from the second chip to one intermediate chip, a second bias coefficient from one intermediate chip to another intermediate chip, and a third bias coefficient from another intermediate chip to the first chip; for example, the first bias coefficient, the second bias coefficient and the third bias coefficient are multiplied to obtain the path bias coefficient.
[0174] After determining the path and the path bias coefficient corresponding to the path, determining the correction value corresponding to the second chip based on at least one path bias coefficient corresponding to the second chip may include:
[0175] Determine a correction value corresponding to the second chip based on one of the at least one path bias coefficient; or,
[0176] A geometric mean is calculated for at least one path bias coefficient, and a correction value corresponding to the second chip is determined by using the value obtained by the geometric mean.
[0177] In this embodiment, for a second chip having at least one path, any path bias coefficient can be used as the correction value of the second chip, or the geometric mean of all path bias coefficients can be calculated to obtain the value as the correction value of the second chip. Optionally, the above Figure 3 Taking the structure shown in the figure as an example, the obtained bias coefficients are: k 21 , k 12 , k 34 , k 43 , k 23 , k 32 , k 14 , k 41 ; From this, we can see that the path bias coefficients corresponding to the two paths corresponding to chip ③ are k 32 *k 21 and k 34 *k 41 , can be based on k 32 *k 21 or k 34 *k 41 As the correction value of chip ③ relative to chip ①, or based on As the correction value of chip ③ relative to chip ①, the correction value determined by geometric mean can achieve better fusion effect because it integrates all bias coefficients of multiple paths; and the correction value from chip ② to chip ① is k 21 , the correction value from chip ④ to chip ① is k 41 .
[0178] The data fusion algorithm of the multi-chip sensor provided in the embodiment of the present disclosure adopts innovative chip bias calculation and correction calculation, effectively solving the problem of data fusion of multi-chip splicing sensors. Through a variety of optional modes, it can flexibly meet the needs of different scenarios and achieve high-quality data fusion.
[0179] like Figure 8 As shown in the above Figure 2 Based on the illustrated embodiment, in another optional embodiment, the brightness-related value is a brightness value; step 204 may include the following steps:
[0180] Step 2045: construct a linear model for the brightness values in two adjacent adjacent regions through linear fitting.
[0181] Optionally, a linear model of a preset structure is determined; parameters in the linear model are determined based on all brightness values in two adjacent adjacent regions; and the parameters are brought into the linear model to obtain a linear model with known parameters.
[0182] Optionally, a quadratic equation can be constructed as a linear model, for example, y=k*x+b, where x represents the brightness value of the pixel in the adjacent area of the chip to be corrected, and y represents the brightness value corresponding to the adjacent chip. When the brightness value of the chip to be corrected needs to be corrected to the brightness value of the adjacent chip, each brightness value x and y is brought into the linear model, and the parameters k and b in the linear model are determined, so as to determine the complete linear model.
[0183] Step 2046, determine the bias coefficient based on the parameters in the linear model.
[0184] In this embodiment, the parameters determined in the linear model can be used as bias coefficients, and the brightness values of all pixels in the pixel areas corresponding to the adjacent chips can be adjusted based on the bias coefficients, so that the brightness between the adjacent chips can be unified. Correspondingly, adjusting the brightness values in the pixel areas corresponding to the chips based on the bias coefficients determined by the linear model can include:
[0185] Determine one chip from the plurality of chips as a first chip and the other chips as second chips;
[0186] Based on the linear models corresponding to every two adjacent chips, determining an updated linear model from each second chip to the first chip;
[0187] Similar to the above embodiment, this embodiment also randomly determines a chip as the first chip (reference chip), and approaches the pixel brightness values of other second chips to the first chip. In this embodiment, the linear model between every two adjacent chips is determined. When the second chip is not adjacent to the first chip, for example, when there is a second chip between the second chip and the first chip, the first linear model between the second chip and the adjacent second chip is first determined, and then the second linear model between the adjacent second chip and the first chip is determined. The first linear model is substituted into the second linear model to obtain an updated linear model. For example, the first linear model is y1=k1*x1+b1, and the second linear model is y2=k2*x2+b2, wherein x2=y1, and after substitution, y2=k2*(k1*x1+b1)+b2=k1*k2*x1+k2*b1+b2; that is, the updated linear model is obtained. By substituting the brightness values in the adjacent areas corresponding to the first chip and the second chip, the values of k1, k2, b1 and b2 can be determined, and the parameters in the updated linear model can be determined.
[0188] The brightness values in the pixel areas corresponding to each second chip are respectively brought into the corresponding updated linear model to obtain a plurality of adjusted brightness values corresponding to the pixel areas.
[0189] By determining the updated linear model, the brightness values of all pixels in the pixel area corresponding to the second chip can be unified with the pixel brightness values in the pixel area corresponding to the first chip and the second chip. In this embodiment, the bias coefficient is calculated by linear regression, and a linear model can be obtained by fitting the data at the junction of adjacent chips, thereby estimating the bias coefficient between chips.
[0190] In some optional embodiments, in order to adjust the brightness value in the pixel area corresponding to the chip using the bias coefficient in non-real time, the method provided in this embodiment may further include:
[0191] The bias coefficient between every two adjacent chips among the multiple chips is stored in a pre-stored database.
[0192] This embodiment stores the calculated bias coefficients in a pre-stored database, that is, determines the bias coefficients between multiple chips in the sensor. When the sensor needs to be used for image acquisition, the bias coefficients in the pre-stored database can be directly called to adjust the brightness value of the image acquired by the sensor to obtain the adjusted image. In this embodiment, the method for obtaining the bias coefficients is not limited, for example, the bias coefficients of the chip determined by a predetermined data set; the bias coefficients of the chip determined based on data corresponding to any scene at a previous moment or the current scene, etc. This embodiment simplifies the real-time correction process and improves the operation efficiency.
[0193] Optionally, since the bias coefficient can be determined in non-real time, that is, the bias coefficient of each chip relative to the adjacent chip is already stored in the prediction database, the method provided in this embodiment may further include:
[0194] In response to the bias coefficient between every two adjacent chips in the plurality of chips included in the pre-stored database;
[0195] The brightness value of at least one adjacent area among the plurality of adjacent areas is adjusted based on the plurality of pre-stored bias coefficients.
[0196] In this embodiment, when there is a bias coefficient in the pre-stored database, when adjusting the pixel brightness value in the pixel area corresponding to the chip, the bias coefficient can be directly used to determine the correction value of each chip relative to the reference chip (the method for determining the correction value based on the bias coefficient can be determined using any of the above embodiments), and then based on the correction value, the pixel brightness value corresponding to each chip is adjusted to be consistent with the reference chip (the adjustment process can refer to any of the above embodiments); or the bias coefficient is directly used to adjust the pixel brightness value corresponding to each chip to be consistent with the reference chip (corresponding to the above embodiment of determining the bias coefficient based on the linear model); according to the needs of the actual application scenario, the sensor data can be corrected in real time or non-real time to meet the real-time requirements of different scenarios. The method provided in this embodiment is highly adaptable, and real-time or non-real-time data correction methods can be selected according to the actual application scenario to meet the real-time requirements of different scenarios.
[0197] Any data fusion method in a sensor provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any data fusion method in a sensor provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any data fusion method in a sensor mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.
[0198] Fig. 9 FIG. 1 is a schematic diagram of the structure of a data fusion device in a sensor provided by an exemplary embodiment of the present disclosure. Fig. 9 As shown, the device provided in this embodiment includes:
[0199] The region determination module 91 is used to determine at least one adjacent region corresponding to each chip based on the positional relationship between the plurality of chips distributed in the array in the sensor, so as to obtain a plurality of adjacent regions; wherein the adjacent region is located adjacent to the pixel regions corresponding to the two chips;
[0200] The bias coefficient determination module 92 is used to determine the bias coefficient between every two adjacent chips in the plurality of chips based on the brightness correlation values in the plurality of adjacent areas in response to the pre-stored database not including the bias coefficient between every two adjacent chips in the plurality of chips; wherein each of the bias coefficients represents a brightness performance error of one chip relative to the other chip in the two adjacent chips;
[0201] A brightness adjustment module 93, configured to adjust the brightness value of at least one pixel region among the plurality of pixel regions based on a plurality of bias coefficients;
[0202] The data fusion module 94 is used to fuse the data output by multiple chips based on the adjusted pixel area.
[0203] The data fusion device in the sensor provided by the above embodiment of the present disclosure determines at least one adjacent area corresponding to each chip based on the positional relationship between multiple chips distributed in an array in the sensor, and obtains multiple adjacent areas; wherein the adjacent area is located adjacent to the pixel areas corresponding to two chips; in response to the fact that the bias coefficient between each two adjacent chips in the multiple chips is not included in the pre-stored database, the bias coefficient between each two adjacent chips in the multiple chips is determined based on the brightness correlation values in the multiple adjacent areas; wherein each bias coefficient represents the brightness performance error of one chip relative to the other chip in the two adjacent chips; based on the multiple bias coefficients, the brightness value of at least one pixel area in the multiple pixel areas is adjusted; based on the adjusted pixel area, the data output by the multiple chips are fused; the embodiment of the present disclosure adjusts the brightness value in the adjacent area by the bias coefficient, so that the brightness values in the pixel areas corresponding to the multiple chips in the sensor are unified, which solves the problem of brightness value jump at the chip junction, makes the pixel brightness transition of multiple chips natural, and improves the quality of sensor output data.
[0204] In some optional embodiments, the area determination module 91 is specifically used to determine at least one adjacent chip corresponding to each of the multiple chips based on the positional relationship between the multiple chips distributed in the array in the sensor; based on a pixel area of a preset size in the pixel area corresponding to each chip that is connected to the pixel area corresponding to the adjacent chip, determine the at least one adjacent area corresponding to the chip to obtain multiple adjacent areas.
[0205] In some optional embodiments, the brightness-related value is a brightness value;
[0206] The bias coefficient determination module 92 includes:
[0207] a brightness sum value determination unit, configured to determine the brightness sum value corresponding to each adjacent area based on the brightness values in the plurality of adjacent areas;
[0208] The first bias coefficient unit is used to determine the bias coefficients corresponding to two adjacent chips based on the brightness and value of two adjacent adjacent areas corresponding to the two adjacent chips.
[0209] Optionally, the sum value determination unit is specifically used to sum the brightness values corresponding to all pixels included in the adjacent area to obtain the brightness sum value corresponding to the adjacent area; or to weight the brightness values corresponding to all pixels included in the adjacent area to obtain the brightness sum value corresponding to the adjacent area.
[0210] In some other optional embodiments, the brightness-related value is pulse data;
[0211] The bias coefficient determination module 92 includes:
[0212] A pulse sum value determination unit, used to determine the pulse sum value corresponding to each adjacent area based on the pulse data in the plurality of adjacent areas;
[0213] The second bias coefficient unit is used to determine the bias coefficients corresponding to two adjacent chips based on the pulse sum values of two adjacent adjacent areas corresponding to the two adjacent chips.
[0214] In some optional embodiments, the brightness adjustment module 93 includes:
[0215] A reference determination unit, used to determine one chip from a plurality of chips as a first chip and the other chips as second chips;
[0216] A correction value unit, used to determine a correction value corresponding to each second chip based on a plurality of bias coefficients corresponding to the plurality of second chips;
[0217] The area adjustment unit is used to adjust the brightness value in the pixel area corresponding to the second chip based on the correction value to obtain at least one adjusted pixel area.
[0218] Optionally, the correction value unit is specifically used to determine the correction value corresponding to the second chip based on the bias coefficient corresponding to the second chip in response to the second chip being adjacent to the first chip; and to determine the correction value corresponding to the second chip based on at least one path bias coefficient corresponding to the second chip in response to the second chip being not adjacent to the first chip.
[0219] Optionally, the brightness adjustment module 93 further includes:
[0220] A path bias determination unit is used to determine at least one path indirectly connected from the second chip to the first chip; wherein each path includes at least one intermediate chip; based on the bias coefficient corresponding to the second chip and the bias coefficient corresponding to each intermediate chip included in each path, determine the path bias coefficient; wherein each path corresponds to a path bias coefficient.
[0221] Optionally, when the correction value unit determines the correction value corresponding to the second chip based on at least one path bias coefficient corresponding to the second chip, it is used to determine the correction value corresponding to the second chip based on one of the at least one path bias coefficients; or, to obtain the geometric mean of at least one path bias coefficient and determine the correction value corresponding to the second chip with the value obtained by the geometric mean.
[0222] In some further optional embodiments, the brightness-related value is a brightness value;
[0223] The bias coefficient determination module 92 includes:
[0224] A linear model unit, used for constructing a linear model for brightness values in two adjacent adjacent regions by linear fitting;
[0225] The third bias coefficient unit is used to determine the bias coefficient based on the parameters in the linear model.
[0226] Optionally, the linear model unit is specifically used to determine a linear model of a preset structure; determine parameters in the linear model based on all brightness values in two adjacent adjacent areas; and bring the parameters into the linear model to obtain a linear model with known parameters.
[0227] Optionally, the brightness adjustment module 93 is specifically used to determine one chip as the first chip and other chips as the second chip from multiple chips; based on the linear model corresponding to every two adjacent chips, determine the updated linear model of each second chip to the first chip; respectively bring the brightness value in the pixel area corresponding to each second chip into the corresponding updated linear model to obtain multiple adjusted brightness values corresponding to the pixel area.
[0228] In some optional embodiments, the device provided in this embodiment may further include:
[0229] A pre-stored coefficient module, used to respond to the bias coefficient between every two adjacent chips in the pre-stored database;
[0230] The brightness adjustment module 93 is used to adjust the brightness value of at least one adjacent area among the multiple adjacent areas based on the pre-stored multiple bias coefficients.
[0231] In some optional embodiments, the device provided in this embodiment may further include:
[0232] The coefficient storage module is used to store the bias coefficient between every two adjacent chips in the plurality of chips into a pre-stored database.
[0233] According to another aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor, and a memory communicatively connected to the processor, and further comprising the data fusion method in a sensor as described in any one of the above embodiments;
[0234] The memory stores computer-executable instructions;
[0235] The processor executes computer-executable instructions stored in the memory to control a data fusion method in the sensor.
[0236] The electronic device provided by the present disclosure may be incorporated into any of the following: pulse cameras, high-speed cameras, audio / video players, navigation devices, fixed-position terminals, entertainment units, smart phones, communication devices, devices in motor vehicles, cameras, sports or wearable cameras, detection equipment, flight equipment, medical equipment, security equipment, etc.
[0237] The electronic device provided by the present disclosure can be applied to any of the following: pulse cameras, high-speed cameras, audio / video players, navigation equipment, fixed-position terminals, entertainment units, smart phones, communication equipment, equipment in motor vehicles, cameras, sports or wearable cameras, detection equipment, flight equipment, medical equipment, security equipment, etc.
[0238] Below, reference Fig.10 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.
[0239] Fig.10 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.
[0240] like Fig.10 As shown, the electronic device includes one or more processors and memory.
[0241] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
[0242] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the data fusion method in the sensor of each embodiment of the present disclosure described above and / or other desired functions.
[0243] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0244] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.
[0245] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0246] Of course, to simplify, Fig.10 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.
[0247] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the data fusion method in the sensor according to various embodiments of the present disclosure described in the above part of this specification.
[0248] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0249] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the data fusion method in a sensor according to various embodiments of the present disclosure described in the above part of this specification.
[0250] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0251] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0252] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0253] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.
[0254] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
[0255] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0256] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0257] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A data fusion method in a sensor, It is characterized in that include: Based on the positional relationship between the plurality of chips distributed in the array in the sensor, at least one adjacent region corresponding to each chip is determined to obtain a plurality of adjacent regions; wherein the adjacent regions are located adjacent to the pixel regions corresponding to two of the chips; In response to the pre-stored database not including the bias coefficient between every two adjacent chips among the plurality of chips, determining the bias coefficient between every two adjacent chips among the plurality of chips based on the brightness correlation values in the plurality of adjacent areas; wherein each of the bias coefficients represents a performance error of one chip relative to the other chip among the two adjacent chips; Based on the plurality of bias coefficients, adjusting the brightness value of at least one pixel area among the plurality of pixel areas; fusing the data output by the plurality of chips based on the adjusted pixel area; If the brightness correlation value is a brightness value; determining the bias coefficient between every two adjacent chips in the plurality of chips based on the brightness correlation values in the plurality of adjacent areas comprises: Over a preset time length, the brightness values corresponding to all pixels included in the adjacent area are accumulated and summed to obtain the brightness sum value corresponding to the adjacent area; or, the brightness values corresponding to all pixels included in the adjacent area are weighted to obtain the brightness sum value corresponding to the adjacent area; the weight value is determined according to the signal-to-noise ratio or perception range of the data at the junction of adjacent chips; Based on the brightness and value of two adjacent adjacent areas corresponding to the two adjacent chips, the bias coefficients corresponding to the two adjacent chips are determined.
2. The method according to claim 1, It is characterized in that The step of determining at least one adjacent region corresponding to each chip based on the positional relationship between the plurality of chips distributed in an array in the sensor to obtain the plurality of adjacent regions comprises: Based on the positional relationship between the plurality of chips distributed in an array in the sensor, determining at least one adjacent chip corresponding to each chip in the plurality of chips; Based on a pixel area of a preset size that is adjacent to a pixel area corresponding to a chip located adjacent to the chip in the pixel area corresponding to each chip, the at least one adjacent area corresponding to the chip is determined to obtain a plurality of adjacent areas.
3. The method according to claim 1, It is characterized in that If the brightness related value is pulse data; The step of determining the bias coefficient between every two adjacent chips in the plurality of chips based on the brightness correlation values in the plurality of adjacent areas comprises: Based on the pulse data in the plurality of adjacent regions, determining a pulse sum value corresponding to each of the adjacent regions; Based on the pulse sum values of two adjacent adjacent areas corresponding to the two adjacent chips, the bias coefficients corresponding to the two adjacent chips are determined.
4. The method according to any one of claims 1 to 3, It is characterized in that The adjusting the brightness value of at least one pixel area among the plurality of pixel areas based on the plurality of bias coefficients comprises: Determine one chip from the plurality of chips as a first chip and the other chips as second chips; Determine a correction value corresponding to each of the second chips based on a plurality of bias coefficients corresponding to the plurality of second chips; The brightness value in the pixel area corresponding to the second chip is adjusted based on the correction value to obtain at least one adjusted pixel area.
5. The method according to claim 4, It is characterized in that The step of determining the correction value corresponding to each second chip based on the plurality of bias coefficients corresponding to the plurality of second chips comprises: In response to the second chip being adjacent to the first chip, determining a correction value corresponding to the second chip based on a bias coefficient corresponding to the second chip; In response to the second chip not being adjacent to the first chip, a correction value corresponding to the second chip is determined based on at least one path bias coefficient corresponding to the second chip.
6. The method according to claim 4, It is characterized in that Before determining the correction value corresponding to the second chip based on at least one path bias coefficient corresponding to the second chip, the method further includes: Determine at least one path indirectly connected from the second chip to the first chip; wherein each of the paths includes at least one intermediate chip; The path bias coefficient is determined based on the bias coefficient corresponding to the second chip and the bias coefficient corresponding to each intermediate chip included in each of the paths; wherein each of the paths corresponds to one path bias coefficient.
7. The method according to claim 6, It is characterized in that The determining, based on at least one path bias coefficient corresponding to the second chip, a correction value corresponding to the second chip includes: determining a correction value corresponding to the second chip based on one of the at least one path bias coefficient; or, A geometric mean is calculated for the at least one path bias coefficient, and a correction value corresponding to the second chip is determined using a value obtained by the geometric mean.
8. The method according to claim 1, It is characterized in that If the brightness related value is a brightness value; The step of determining the bias coefficient between every two adjacent chips in the plurality of chips based on the brightness correlation values in the plurality of adjacent areas comprises: Constructing a linear model for the brightness values in two adjacent adjacent regions by linear fitting; The bias coefficient is determined based on the parameters in the linear model.
9. The method according to claim 8, It is characterized in that The step of constructing a linear model for the brightness values in two adjacent adjacent regions by linear fitting includes: Determine the linear model of the preset structure; Determining parameters in the linear model based on all brightness values in two adjacent adjacent regions; Substitute the parameters into the linear model to obtain the linear model with known parameters.
10. The method according to claim 8, It is characterized in that The adjusting the brightness value of at least one pixel area among the plurality of pixel areas based on the plurality of bias coefficients comprises: Determine one chip from the plurality of chips as a first chip and the other chips as second chips; Based on the linear models corresponding to every two adjacent chips, determining an updated linear model from each second chip to the first chip; The brightness values in the pixel areas corresponding to each of the second chips are respectively brought into the corresponding updated linear model to obtain a plurality of adjusted brightness values corresponding to the pixel areas.
11. The method according to any one of claims 1 to 3, It is characterized in that Also includes: In response to the pre-stored database including a bias coefficient between every two adjacent chips among the plurality of chips; The brightness value of at least one adjacent area among the plurality of adjacent areas is adjusted based on the plurality of pre-stored bias coefficients.
12. The method according to any one of claims 1 to 3, It is characterized in that Also includes: The bias coefficient between every two adjacent chips in the plurality of chips is stored in the pre-stored database.
13. A data fusion device in a sensor, It is characterized in that include: A region determination module, used to determine at least one adjacent region corresponding to each chip based on the positional relationship between the plurality of chips distributed in an array in the sensor, so as to obtain a plurality of adjacent regions; wherein the adjacent region is located adjacent to the pixel regions corresponding to two of the chips; A bias coefficient determination module, configured to determine the bias coefficient between every two adjacent chips among the plurality of chips based on the brightness correlation values in the plurality of adjacent areas in response to the pre-stored database not including the bias coefficient between every two adjacent chips among the plurality of chips; wherein each of the bias coefficients represents a performance error of one chip relative to the other chip among the two adjacent chips; A brightness adjustment module, configured to adjust a brightness value of at least one pixel area among the plurality of pixel areas based on the plurality of bias coefficients; A data fusion module, used for fusing the data output by the plurality of chips based on the adjusted pixel area; If the brightness-related value is a brightness value, the bias coefficient determination module includes: a brightness sum value determination unit, which is used to accumulate and sum the brightness values corresponding to all pixels included in the adjacent area over a preset time length to obtain the brightness sum value corresponding to the adjacent area; or, weightedly process the brightness values corresponding to all pixels included in the adjacent area to obtain the brightness sum value corresponding to the adjacent area; the weight value is determined according to the signal-to-noise ratio or perception range of the data at the junction of adjacent chips; and a first bias coefficient unit, which is used to determine the bias coefficients corresponding to two adjacent chips based on the brightness sum values of two adjacent adjacent areas corresponding to the two adjacent chips.
14. An electronic device, It is characterized in that include: A processor, and a memory in communication with the processor, further comprising the data fusion method in the sensor according to any one of claims 1 to 12; The memory stores computer-executable instructions; The processor executes computer-executable instructions stored in the memory to control a data fusion method in the sensor.
15. The electronic device according to claim 14, It is characterized in that The electronic device is included in any of the following: pulse camera, high-speed camera, audio / video player, navigation equipment, fixed position terminal, entertainment unit, smart phone, communication equipment, equipment in motor vehicle, camera, sports or wearable camera, detection equipment, flight equipment, medical equipment, security equipment.
16. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, and when the instructions are executed, the computer executes the data fusion method in the sensor according to any one of claims 1 to 12.
