Image feature fusion method and device and machine vision system

Abstract

The image feature fusion method is applied to a machine vision system, and the machine vision system at least comprises a first lens and a visible light lens. The resolution of the first lens is lower than that of the visible light lens. The method comprises the following steps: acquiring multiple mode images for the same object; the multi-mode image comprises a first image and a visible light image; the first image is an image in a first video collected by a first lens, and the visible light image is an image in a visible light video collected by a visible light lens; according to registration of the first image and the visible light image, a first projective transformation relation is obtained; features and a first position of the target object are extracted from the first image, the features of the target object comprise the contour of the target object, and the first position comprises coordinates of pixel points of the contour of the target object on the first image; feature level fusion is carried out on the first image and the visible light image, the outline of the target object is displayed in an enhanced mode at a second position on the visible light image, and the second position and the first position conform to the first projective transformation relation.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an image feature fusion method, device and machine vision system. Background Art

[0002] In low visibility environments such as at night and in heavy fog, visible light lenses cannot clearly perceive people, vehicles, and objects in the outside world, and the images obtained are generally blurry, but infrared lenses can clearly obtain infrared images of people, vehicles, and objects. The dual-modal fusion of visible light images and infrared images can achieve information complementarity, so that the fused image contains more comprehensive and rich information. The fused image is more in line with the visual characteristics of humans or machines, and is more conducive to further analysis and processing of the image and automatic target recognition. The fusion of infrared images and visible light images retains the thermal radiation information of infrared images and the texture information of visible light images, and is widely used in the field of night vision and other low-recognition scenes.

[0003] The most widely studied infrared and visible light image fusion methods are based on multi-scale decomposition, sparse representation, saliency, deep learning, etc. There are also simple two-scale image fusion (TIF) methods. Taking TIF as an example, although this algorithm is widely used, the fused image is fuzzy and has a large difference from the actual image of the real environment. Under certain conditions, it will produce an uncomfortable experience, such as a ghost eye effect on a person's face image, which will scare the operator. Summary of the invention

[0004] An embodiment of the present application provides an image feature fusion method, which is applied to a machine vision system, wherein the machine vision system includes at least a first lens and a visible light lens; the resolution of the first lens is lower than that of the visible light lens, and the method includes: acquiring multiple mode images of the same object; the multiple mode images include a first image and a visible light image; the first image is an image in a first video acquired by the first lens, and the visible light image is an image in a visible light video acquired by the visible light lens; a first projective transformation relationship is obtained based on registration of the first image and the visible light image; a feature and a first position of a target object are extracted from the first image, wherein the feature includes a contour of the target object, and the first position includes the coordinates of pixel points of the contour of the target object on the first image; the first image and the visible light image are fused at the feature level, and the contour of the target object is displayed at a second position on the visible light image, and the second position and the first position conform to the first projective transformation relationship.

[0005] Therefore, the image feature fusion method proposed in the embodiment of the present application can present the complete outline of the target object on the visible light image in the scene of reflection of street lights at night or strong reflection of opposite objects. In scenes with insufficient light such as late at night, light fog, rainy weather, etc., it can improve the vehicle equipment's ability to recognize people, vehicles, and objects on the road and improve the operational safety of the system.

[0006] In a possible implementation, the machine vision system further includes a second lens, the second lens has the same resolution as the first lens, the multiple mode images further include a second image captured by the second lens, and the method further includes: obtaining a second projective transformation relationship based on registration of the second image and the visible light image; extracting features and a third position of the target object from the second image, the third position including coordinates of pixel points of the outline of the target object on the second image; performing pixel-level fusion calculations on pixel points within the third position of the second image and pixel points within the second position on the visible light image to obtain new pixel values; the third position and the second position conform to a second projective transformation relationship; and using the new pixel value as the pixel value of the pixel point within the outline of the target object on the visible light image.

[0007] Therefore, the image feature fusion method proposed in the embodiment of the present application improves the visual comfort of the image seen by the human eye and eliminates the "ghost eye" effect by fusion of the feature level and pixel level of the dual infrared images and the visible light image in the scene where the visible light is not clear. It can also enhance the recognition ability of the visual system by accurately fusing the depth information and the image detail information.

[0008] In a possible implementation, a first projective transformation relationship is obtained based on the registration of the first image and the visible light image, including: collecting multiple sets of image pairs of first images and visible light images for the same object; and obtaining the first projective transformation relationship based on coordinate calculation of multiple points on the multiple sets of image pairs of first images and visible light images.

[0009] Therefore, the image feature fusion method proposed in the embodiment of the present application aligns the infrared image and the visible light image to ensure that the positions of the pixels of the two images are consistent after overlapping, thereby preparing for fusion.

[0010] In a possible implementation, extracting features and a first position of a target object from a first image includes: inputting the first image into a neural network recognition model, and obtaining contour features and a first position of the target object according to an output of the neural network recognition model; wherein the neural network recognition model includes a FYOLACT network.

[0011] Therefore, the image feature fusion method proposed in the embodiment of the present application segments the first image and interprets the structure of the image with multiple resolutions. The bottom of the image pyramid is a high-resolution representation of the image to be processed, while the top is a low-resolution approximation. The higher the level, the smaller the image and the lower the resolution. By downsampling the first image in stages, the most objects of various scales on the first image can be detected.

[0012] In a possible implementation, the contour features and the first position of the target object are obtained according to the output of the neural network recognition model, including: the FYOLACT network divides the first image into images of different sizes to obtain an image pyramid; the bottom layer of the image pyramid is a high-resolution representation of the first image, and the top layer of the image pyramid is a low-resolution representation of the first image, and the lower the resolution, the smaller the corresponding image and the higher the level; each layer of the image of the image pyramid is convolved and differentially calculated through a convolutional network model, and the feature distribution of each layer of the image is output; according to the feature distribution of each layer of the image, the feature pyramid network is used in combination with the feedforward calculation of CNN to predict and output the contour features of the target object and the coordinates of the corresponding pixel points.

[0013] Therefore, the image feature fusion method proposed in the embodiment of the present application performs up-sampling and Gaussian convolution on the feature distribution of the bottom layer, and fuses it with the bottom layer feature to obtain high-resolution and strong semantic features, thereby enhancing the extraction of target object features. The feedforward calculation using the FPN algorithm + CNN can pass down the positioning features of the high layers of the infrared image pyramid and supplement the features of the lower layers, so that significant target features can be obtained, which is conducive to the category and position detection of small targets.

[0014] In a possible implementation, the feature also includes one or a combination of the following: category feature, morphological feature, motion feature and identity feature of the target object, and the first position also includes one or a combination of the following: the coordinates of the pixel points corresponding to the category feature, morphological feature, motion feature and identity feature on the first image.

[0015] Therefore, the image feature fusion method proposed in the embodiment of the present application can identify changes in people, vehicles, and objects within the camera field of view in scenes with insufficient light, such as darkness, thereby improving recognition accuracy.

[0016] In a possible implementation, the method further includes: performing image quality enhancement processing on the first image and / or the second image, where the image quality enhancement processing includes video noise reduction, deblurring, sharpening and / or de-jittering.

[0017] Therefore, the image feature fusion method proposed in the embodiment of the present application can improve the visual comfort of the image seen by the human eye.

[0018] In a possible implementation, the method further includes: performing video super-resolution processing on the first image and / or the second image.

[0019] Therefore, the image feature fusion method proposed in the embodiment of the present application can improve the chromaticity, brightness and resolution, and improve the visual clarity of the image seen by the human eye.

[0020] In a possible implementation, the method further includes: performing disparity calculation based on the first image and the second image, and obtaining a disparity grayscale image based on difference encoding; obtaining the distance of the target object based on the disparity grayscale image and the fused image, wherein the fused image includes an image obtained by feature-level fusion or an image obtained by pixel-level fusion.

[0021] Therefore, the image feature fusion method proposed in the embodiment of the present application can accurately measure and judge the distance to the target object.

[0022] In a possible implementation, the first lens and the second lens are infrared lenses, so that infrared images of people, vehicles, and objects can be clearly obtained in low-visibility environments such as night and fog.

[0023] In a possible implementation, the method further includes: the machine vision system includes a laser detection device; the laser detection device transmits / receives a laser signal, and the method includes: fusing the returned laser signal reflected by the target object with the visible light image to obtain the distance of the real target.

[0024] In a possible implementation, the method further includes: the machine vision system includes a millimeter wave radar device; the millimeter wave radar device transmits / receives a millimeter wave radar signal, and the method includes: fusing the returned millimeter wave radar signal reflected by the target object with the visible light image to obtain the true distance of the target object.

[0025] In the second aspect, the embodiment of the present application provides a device for image feature-level fusion, which is applied to a machine vision system, wherein the machine vision system includes at least a first lens and a visible light lens; the resolution of the first lens is lower than that of the visible light lens, and the device includes: an image acquisition module, which is used to acquire multiple mode images of the same object; the multiple mode images include a first image and a visible light image; the first image is an image in a first video acquired by the first lens, and the visible light image is an image in a visible light video acquired by the visible light lens; a first image processing module, which is used to obtain a first projective transformation relationship according to the registration of the first image and the visible light image; extracting the features and the first position of the target object from the first image, wherein the features include the contour of the target object, and the first position includes the coordinates of the pixel points of the contour on the first image; a first image fusion module, which is used to perform feature-level fusion on the first image and the visible light image; a first display module, which is used to enhance the display of the contour of the target object at a second position on the visible light image, and the second position and the first position conform to the first projective transformation relationship. Its beneficial effects are as described in the first aspect, and will not be repeated here.

[0026] In a possible implementation, the machine vision system further includes a second lens, the second lens has the same resolution as the first lens, the multiple mode images further include a second image captured by the second lens, and the device further includes: coordinates of pixel points of the contour; a second image fusion module, used to perform pixel-level fusion calculations on pixel points in a third position of the second image and pixel points in a second position on the visible light image to obtain new pixel values; the third position and the second position conform to a second projective transformation relationship; and a second display module, used to display the new pixel values ​​of the pixel points within the contour of the target object on the visible light image.

[0027] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute any method of the first aspect. The beneficial effects are as described in the first aspect and will not be repeated here.

[0028] In a fourth aspect, an embodiment of the present application provides a machine vision system, the machine vision system comprising a first lens, a second lens, a visible light lens, and an image feature level fusion device; wherein the resolution of the first lens is lower than that of the visible light lens; the second lens has the same resolution as the first lens; the first lens is used to capture a first video; the second lens is used to capture a second video; the visible light lens is used to capture a visible light video; and the image feature level fusion device is used to perform any method as in the first aspect. Its beneficial effects are as described in the first aspect and will not be repeated here.

[0029] In a possible implementation, the machine vision system further includes a laser detection device; the image feature level fusion device fuses the laser signal returned by the laser detection device with the visible light image to obtain the distance of the real target corresponding to the target object on the visible light image.

[0030] In a possible implementation, the machine vision system also includes a millimeter wave radar device; the image feature level fusion device fuses the millimeter wave signal returned by the millimeter wave radar device with the visible light image to obtain the distance of the real target corresponding to the target object on the visible light image. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this application specification, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] The following is a brief introduction to the drawings required for use in the embodiments of the present application or the description of the prior art.

[0033] Figure 1 A hardware architecture diagram of a machine vision system proposed in an embodiment of the present application;

[0034] Figure 2 A software system architecture diagram of an image fusion method proposed in an embodiment of the present application;

[0035] Figure 3 A flowchart of an image fusion method according to an embodiment of the present application;

[0036] Figure 4 A flow chart of the registration algorithm of the image feature fusion method proposed in the embodiment of the present application;

[0037] Figure 5 This is a schematic diagram of the algorithm of the FYOLACT network in the image feature fusion method proposed in the embodiment of the present application;

[0038] Figure 6 A flowchart of the fusion of dual infrared images and visible light images in the image feature fusion method proposed in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of a scene of the image feature fusion method proposed in Example 1 of the present application;

[0040] Figure 8 This is a schematic diagram of the image feature fusion method proposed in Example 3 of the present application for measuring the distance of a target object. DETAILED DESCRIPTION

[0041] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0042] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, and multiple terminals refers to two or more terminals.

[0043] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0044] In the description of the embodiments of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] In the description of the embodiments of the present application, the terms "first\second\third, etc." or module A, module B, module C, etc. are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that the specific order or sequence can be interchanged where permitted so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0046] Relevant terms involved in the embodiments of this application:

[0047] Homography matrix (H matrix), the homogeneous coordinates of point pairs on two images of different perspectives can be expressed by a projective transformation, such as: x1 = H*x2, the H matrix in the formula is the homography matrix. In computer vision, solving the homography matrix is ​​of great significance in the process of lens calibration. The homography matrix contains the lens's intrinsic parameter matrix, rotation vector and translation vector.

[0048] In the description of the embodiments of the present application, the numbers representing the steps, such as S110, S120, etc., do not necessarily mean that the steps will be executed in this manner. If permitted, the order of the previous and next steps can be interchanged, or they can be executed simultaneously.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] Among the related technical solutions, the first one adopts a two-scale image fusion (TIF) method, which uses different algorithms to align the infrared image and the visible light image, and then fuses the pixels using a unified weight ratio to generate an image.

[0051] The second method is to directly separate the base layer and detail layer of the infrared image and the visible light image, and then fuse the infrared image and the visible light image in a constant ratio to generate an image.

[0052] The above two solutions use a single infrared lens to realize the recognition of people, vehicles and objects under dark light, foggy or reflective conditions, and the resolution is low. Although the visible light image generated by this type of fusion has contours, the details are not clear. Especially when processing close-up portraits under dark light conditions, the task image after direct fusion has a "ghost face" effect, which will cause discomfort to users and a poor experience. In addition, when the fusion technology is actually used, the images taken by the infrared lens and the visible light lens are required to be strictly matched, which places high requirements on the accuracy of the alignment of the infrared lens and the visible light lens.

[0053] In view of this, the embodiments of the present application propose an image feature fusion method, device and machine vision system capable of identifying targets in multi-source videos, wherein the targets may be people, vehicles and / or objects that are unclear / unrecognizable in visible light images. The image fusion method proposed in the embodiments of the present application can restore real scenes in dark light, foggy or reflective conditions without visually impacting users, thereby improving user experience.

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0055] Figure 1 This is a hardware architecture diagram of the machine vision system proposed in the embodiment of this application. Figure 1 As shown, the hardware devices of the system include one or more infrared lenses 11, visible light lenses 12, computing devices 13 and user terminals 14, etc.

[0056] Among them, one or more infrared lenses 11 collect infrared videos, and the visible light lens 12 collects visible light videos of the same scene.

[0057] The computing device 13 jointly aligns the first image captured by the first infrared lens and the visible light image in the visible light video to obtain a first projective transformation relationship; extracts features and a first position of the target object from the first image, wherein the features of the target object include the outline of the target object, and the first position includes the coordinates of the pixel points of the outline of the target object on the first image; performs feature-level fusion on the first image and the visible light image, and displays the outline of the target object at a second position on the visible light image, wherein the second position and the first position conform to the first projective transformation relationship.

[0058] The computing device 13 also registers the second image and the visible light image captured by the second infrared lens to obtain a second projective transformation relationship; extracts the features and the third position of the target object from the second image, the third position including the coordinates of the pixel points of the outline of the target object on the second image; performs pixel-level fusion calculation on the pixel points in the third position of the second image and the pixel points in the second position on the visible light image to obtain a new pixel value; the third position and the second position conform to the second projective transformation relationship; the new pixel value is used as the pixel value of the pixel point in the outline of the target object on the visible light image to obtain a fused image. The first infrared lens and the second infrared lens can be respectively recorded as the first lens and the second lens.

[0059] The computing device 13 performs disparity calculation based on the first image and the second image, encodes the difference into a disparity grayscale image, and performs secondary fusion with the fused image based on the disparity grayscale image to obtain the distance of the target object.

[0060] It should be understood that parallax is the difference in the x-coordinates of the same spatial point in the two cameras, which can be encoded into a disparity grayscale map. The disparity grayscale map can reflect the distance. The closer to the lens, the brighter the grayscale.

[0061] The computing device 13 combines images from fusion sources such as visible light and infrared light, detects, enhances, and fuses them through algorithms, and identifies information and / or distances of people, vehicles, and objects at low recognition levels. It provides behavioral decisions for the autonomous driving vehicle equipment, such as automatically initiating avoidance actions when a person is detected.

[0062] The user terminal 14 includes vehicle equipment, mobile portable devices, mobile camera equipment, display equipment, and a next-level decision terminal. The user terminal 14 displays a fused image, which has clear image quality and is close to the actual image of the real environment. It can also identify the target under certain low-recognition conditions such as dark light, fog, and exposure, reflecting the distance of the target, and will not produce an uncomfortable experience.

[0063] The above is an introduction to the hardware architecture and application scenarios of the image fusion method provided in the embodiment of the present application. Next, based on the above content, an image fusion method provided in the embodiment of the present application is introduced. For the introduction to the concepts, units, etc. involved in the following content, please refer to the above.

[0064] Figure 2 This is a software system architecture diagram of an image fusion method proposed in an embodiment of the present application. Figure 2 As shown, based on Figure 1 The hardware architecture of the machine vision system shown in the figure identifies the contours of the target objects in the multi-source videos, extracts their corresponding features, accurately fuses the multi-source videos, and visualizes the images. The system architecture of the image feature fusion method includes:

[0065] The image acquisition module 21 is used to acquire multiple mode images of the same object; the multiple mode images include a first image, a second image and a visible light image; the first image is an image in a first video acquired by a first infrared lens, the second image is an image in a second video acquired by a second infrared lens, and the visible light image is an image in a visible light video acquired by a visible light lens. The pixel (resolution) of the visible light video is greater than the pixel (resolution) of the first infrared video or the pixel (resolution) of the second infrared video, and the focal length is the same. The first infrared lens and the second infrared lens can be infrared lenses on the left and right sides of the vehicle equipment.

[0066] In the driving detection of vehicle equipment, the reflection of street lights at night or the strong reflection of opposite objects will cause the visible light image to be incomplete, the contrast is not high enough, and it is impossible to capture all features.

[0067] In an embodiment of the present application, a visible light lens and a polarizing film may be used to capture a visible light image to reduce the influence of reflections, thereby achieving the goal of capturing all features of the target object.

[0068] In some feasible implementations, the multiple mode images also include laser images and millimeter wave radar images, the pixels (resolution) of visible light video > the pixels (resolution) of infrared video > the pixels of lidar video, and the focal lengths are the same.

[0069] The image processing module 22 is used to extract the features of the target object and the addresses of the corresponding pixel points from the low-resolution image; determine the features of the target object in the low-resolution image, the features of the target object include contour, shape, motion or stillness, etc.

[0070] The image processing module 22 includes a feature recognition unit, an image registration unit, a disparity calculation unit, an image enhancement unit and / or a super-resolution technology unit.

[0071] The feature recognition unit inputs the first image or the second image into a trained neural network recognition model, outputs the features of the target object, and performs feature recognition of the target object. The features of the target object include the contour features of the target object.

[0072] The image registration unit obtains a first projective transformation relationship according to the registration of the first image and the visible light image; and / or obtains a second projective transformation relationship according to the registration of the second image and the visible light image.

[0073] The disparity calculation unit calculates the disparity based on the first image and the second image, and obtains a disparity grayscale image based on the difference encoding. The disparity is the difference in the x-coordinates of the same spatial point in the two cameras. The disparity grayscale image is used to reflect the distance of the target object. The closer to the lens, the brighter the grayscale.

[0074] In some optional embodiments, the image processing module 22 also includes an image enhancement unit, which performs image quality enhancement processing such as video noise reduction, deblurring, sharpening, and de-jittering on the first image and / or the second image and the visible light image.

[0075] In some optional implementations, the image processing module 22 further includes a super-resolution technology unit, which performs video super-resolution processing on the first image and / or the second image and the visible light image to improve the resolution of the image.

[0076] Exemplarily, the process of video super-resolution processing includes first decoding the infrared image to obtain a low-resolution infrared image, then decomposing the brightness and chrominance signals, enhancing the brightness and chrominance of the decomposed infrared image respectively, super-reconstructing the enhanced brightness image, and up-sampling the enhanced chrominance signal to obtain chrominance and brightness images with improved resolution. Finally, the brightness and chrominance signals are merged to finally display the infrared image with improved resolution.

[0077] The image fusion module 23 includes a feature-level fusion unit, a pixel-level fusion unit and a distance measurement unit.

[0078] The feature-level fusion unit extracts the features and the first position of the target object from the first image. The features of the target object include the outline of the target object. The first position includes the coordinates of the pixel points of the outline of the target object on the first image. The visible light image is used as the background. According to the first projection transformation relationship, the second position corresponding to the features of the target object is calculated on the visible light image. The outline of the target object is displayed at the second position on the visible light image to obtain the first fused image. This improves the device's ability to recognize the content of the visible light image.

[0079] The pixel-level fusion unit obtains the second projective transformation relationship based on the registration of the second image and the visible light image; extracts the features and the third position of the target object from the second image, wherein the features of the target object include the outline of the target object, and the third position includes the coordinates of the pixel points of the outline of the target object on the second image; performs pixel-level fusion calculation on the pixel points in the third position of the second image and the pixel points in the second position on the visible light image to obtain a new pixel value; the third position and the second position conform to the second projective transformation relationship; updates the pixel values ​​of the pixel points in the outline of the target object on the visible light image to the new pixel values ​​to obtain a second fused image. This improves the recognizability and comfort of the visible light image content.

[0080] For example, the character border can be extracted through the infrared image, and then the infrared image and the visible light image are jointly registered. Finally, the coordinate position corresponding to the border is calculated on the visible light image and highlighted to enhance the device's ability to recognize character images.

[0081] The distance measuring unit calculates the distance of the target object according to the disparity grayscale image and the characteristics of the target object, and indicates the distance of the real target corresponding to the target object on the visible light image through the brightness value of the grayscale of the target image.

[0082] In some feasible implementations, the returned laser signal reflected by the target object may be fused with the visible light image to obtain the distance of the real target corresponding to the target object on the visible light image.

[0083] Laser detection equipment uses laser to scan the target object and records the returned "image" of the light intensity and phase of each point on the target on a film. The laser image records all the light information of the target object, and can see the blocked original object and reconstruct the real object.

[0084] In some feasible implementations, the received millimeter-wave radar echo signal may be fused with the visible light image to obtain the distance of the real target corresponding to the target on the visible light image.

[0085] The application module 24 includes a visualization unit, a recognition unit and an action decision unit.

[0086] The visualization unit displays the first fused image and / or the second fused image on the user terminal.

[0087] The recognition unit recognizes the identity of a person based on feature information of the target object in the first fused image and / or the second fused image, such as bones, when visible light cannot be used for recognition.

[0088] The action decision unit gives the next level of action decision for autonomous driving based on the distance between the target object in the visible light image and the real target.

[0089] In some optional implementations, the application module 24 aligns the distance based on the configured laser or millimeter wave radar, performs dynamic measurement, and marks the target object.

[0090] Figure 3 FIG. 1 is a flow chart of an image fusion method according to an embodiment of the present application. Figure 3 As shown, the following steps are included: S31, collecting multiple mode images for the same object; the multiple mode images include a first image and a visible light image; the first image is an image in a first video collected by a first lens, and the visible light image is an image in a visible light video collected by a visible light lens; S32, obtaining a first projective transformation relationship based on the registration of the first image and the visible light image; S33, extracting the characteristics and the first position of the target object from the first image, the characteristics of the target object including the outline of the target object, and the first position including the coordinates of the pixel points of the outline of the target object on the first image; S34, performing feature-level fusion on the first image and the visible light image, and displaying the outline of the target object at a second position on the visible light image, and the second position and the first position conform to the first projective transformation relationship.

[0091] The above steps are further described below in conjunction with the attached specific implementation methods.

[0092] S31, collecting multiple mode images for the same object, wherein the multiple mode images include a first image and a visible light image; the first image is an image in a first video collected by a first lens, and the visible light image is an image in a visible light video collected by a visible light lens. The pixel (resolution) of the visible light video is greater than the pixel (resolution) of the infrared video, and the focal lengths are the same.

[0093] In some feasible implementations, the multiple mode images further include a second image captured by a second lens, and the second lens has the same resolution as the first lens.

[0094] In scenes with insufficient light, such as late at night, in mist, or in rainy weather, multiple mode images can be processed separately.

[0095] For visible light images, polarizing films can be used to reduce the impact of reflections.

[0096] For infrared images and / or visible light images, image quality enhancement processing such as video noise reduction, deblurring, sharpening, and de-shaking can be performed.

[0097] In some feasible implementations, video super-resolution processing may be performed on the infrared image to improve the resolution of the infrared image or the right infrared image.

[0098] S32, obtaining a first projective transformation relationship according to registration of the first image and the visible light image.

[0099] In some feasible implementations, multiple sets of image pairs of first images and visible light images may be collected for the same object; and the first projective transformation relationship may be obtained by calculating the coordinates of multiple points on the multiple sets of image pairs of first images and visible light images.

[0100] In a dual infrared lens scenario, a second projective transformation relationship is obtained by registering the second image with the visible light image.

[0101] In some feasible implementations, multiple sets of image pairs of second images and visible light images may be collected for the same object; and the second projective transformation relationship may be obtained by calculating the coordinates of multiple points on the multiple sets of image pairs of second images and visible light images.

[0102] S33, extracting features and a first position of the target object from the first image, wherein the features of the target object include a contour of the target object, and the first position includes coordinates of pixels of the contour of the target object on the first image.

[0103] In some feasible implementations, the characteristics of the target object also include category characteristics, morphological characteristics, action characteristics, and identity characteristics.

[0104] In some feasible implementations, the first and / or second image may be input into a neural network recognition model, and contour features of the target object may be obtained according to an output of the neural network recognition model.

[0105] The neural network recognition model can adopt the FYOLACT network. The first and / or second image can be input into the trained FYOLACT network to output the contour features and pixel coordinates of the target object to improve the subsequent fusion accuracy and perform target object recognition and ranging.

[0106] S34, performing feature-level fusion on the first image and the visible light image, and displaying the outline of the target object at a second position on the visible light image, wherein the second position and the first position conform to a first projective transformation relationship.

[0107] In some implementable embodiments, feature-level fusion can be performed on the first image and the visible light image. Before fusion, preprocessing operations such as registration are performed on the visible light and infrared data. The visible light image is used as the background, the contour of the target object is extracted from the infrared image, and then the coordinate positions corresponding to the contour are calculated on the visible light image to mark the contour of the target object in the visible light image, thereby improving the device's recognition ability for the content of the visible light image.

[0108] In some implementable embodiments, pixel-level fusion can be performed on the second image and the visible light image. Before fusion, preprocessing operations such as registration are performed on the visible light and infrared data. Then, the pixel values of the corresponding points of the second infrared image and the visible light image are combined and calculated through a certain strategy to obtain a new pixel value. After the pixel values of each point are fused, a visualized image is formed, thereby improving the recognizability of the content of the visible light image.

[0109] Figure 4 The flowchart of the registration algorithm for the image feature fusion method proposed in the embodiment of the present application. It can be understood that this algorithm can be executed by any model, device, platform, or device cluster with computing and processing capabilities. Such as Figure 4 As shown, the registration algorithm in step S32 registers the infrared image and the visible light image to ensure that the positions of the pixels of the two images overlap, thereby preparing for fusion. This step mainly uses the homography matrix to complete the joint registration and calculates the address of the pixel points on the infrared image in the visible light image. The process of the dual-lens registration algorithm in step S32 is as follows:

[0110] S321, Obtain M groups of image data collected by the first infrared lens, the second infrared lens, and the visible light (RGB) lens for the same object.

[0111] Exemplarily, the number of image pairs is 30 groups, that is, the left infrared lens, the right infrared lens, and the visible light lens each collect 30 images.

[0112] S322, Determine whether the collected image data is greater than N groups, where N < M. If the judgment result is yes, execute steps S327 - S328; if the judgment result is no, execute steps S323 - S326.

[0113] Exemplarily, the value of N is 20. Determine whether the collected image data is greater than 20 groups. If the judgment result is yes, execute steps S327 - S328; if the judgment result is no, execute steps S323 - S326.

[0114] S323, Take the coordinates of multiple pairs of feature pixel points on the first image and the visible light image respectively. Exemplarily, take the coordinates of 4 pairs of feature pixel points on the first image and the visible light image pair respectively.

[0115] S324, calculating a first H matrix from the RGB lens to the first infrared lens according to the coordinates of the plurality of pairs of feature points of the first image pair. The first H matrix includes an intrinsic parameter matrix from the RGB lens to the left infrared lens, a rotation vector, and a translation vector.

[0116] S325, respectively obtaining coordinates of a plurality of feature points on the second image pair of the second red image and the visible light image.

[0117] S326, calculating a second H matrix from the second infrared lens to the RGB lens according to the coordinates of the plurality of pairs of feature points of the second image pair. The second H matrix includes an intrinsic parameter matrix from the RGB lens to the right infrared lens, a rotation vector, and a translation vector.

[0118] S327, calculating the average values ​​of the N groups of first H matrices to obtain a first mean matrix H1, where the first mean matrix H1 is a first projective transformation relationship from the RGB lens to the first infrared lens.

[0119] S328, calculating the average values ​​of the N groups of second H matrices to obtain a second mean matrix H2, where the second mean matrix H2 is a second projective transformation relationship from the RGB lens to the second infrared lens.

[0120] S329, perform a registration test on the first image pair in the remaining (MN) groups of image data according to the first mean matrix, perform a registration verification on the second image pair in the remaining (MN) groups of image data according to the second mean matrix, and output the registration result. If the registration result meets the requirements, end the registration. If the registration result does not meet the requirements, return to step S321 to re-register.

[0121] Among them, the registration result should satisfy: x1=H1*x2, x2=H2*x3; that is, the RGB image to the first image conforms to the first projective transformation relationship, and the RGB image to the second image conforms to the second projective transformation relationship.

[0122] When no distinction is needed, in the following embodiments, the first image and the second image may be collectively referred to as infrared images.

[0123] Figure 5 This is a schematic diagram of the algorithm of the FYOLACT network in the image feature fusion method proposed in the embodiment of this application. Figure 5 As shown, in step S33, the FYOLACT network uses a pyramid algorithm and a convolutional neural networks (CNN) difference algorithm to obtain the contour of the target object. The process of step S33 is as follows:

[0124] S331, using a pyramid algorithm to generate an image pyramid, the image pyramid includes images of different sizes, each image includes different features.

[0125] In order to detect as many large and small objects as possible on the first image, the first image may be downsampled in stages, and an image pyramid may be obtained using a pyramid algorithm.

[0126] Image pyramid is a multi-scale representation of an image. It is mainly used for image segmentation and is a structure that interprets images at multiple resolutions. The bottom of the image pyramid is a high-resolution representation of the image to be processed, while the top is a low-resolution approximation. The higher the level, the smaller the image and the lower the resolution.

[0127] In the embodiment of the present application, each layer of the image pyramid is obtained by sequentially downsampling the infrared image until a certain termination condition is reached.

[0128] S332, each layer of the image pyramid is subtracted from the upper layer of the image and upsampled and convolved by the convolutional network model CNN to obtain a series of difference images and corresponding multi-scale feature distributions.

[0129] The use of pyramid algorithm can avoid the loss of some high-frequency detail information of infrared images after convolution and downsampling operations.

[0130] In some practicable implementations, each layer of the convolutional network model CNN outputs a feature distribution (feature map).

[0131] It can be understood that the feature map at the bottom contains more positioning features, while the feature map at the top contains more target features. The features of the target object are used to identify the category of the target object, such as people, cars, objects, faces, bones, etc.

[0132] S333 predicts the features and positions of the target objects based on the multi-scale feature distribution using feature pyramid networks (FPN) combined with CNN feedforward calculation.

[0133] The computational path of the feature pyramid network includes bottom-up, top-down, and lateral connections.

[0134] Specifically, the backbone network of FPN is ResNet, which uses a bottom-up path to extract multi-scale feature distribution. The feedforward calculation of CNN is a top-down path. The multi-scale feature distribution is calculated through top-down convolution kernels to integrate more target features and positioning features. Finally, the horizontal connection calculation path is used to fuse the feature distribution.

[0135] In some possible implementations, such as Figure 6As shown in the figure, the two layers of features connected horizontally have the same spatial size, and can use the underlying positioning details to predict the position. The 1×1 conv is used to change the number of channels of the feature map.

[0136] In some achievable implementations, feature pyramid networks (FPNs) upsample and Gaussian convolve the bottom-level feature distribution and fuse it with the bottom-level features to obtain high-resolution, strong semantic features, thereby enhancing feature extraction. These high-resolution, strong semantic features include the features and positions of the target object.

[0137] The embodiment of the present application adopts the feedforward calculation of the FPN algorithm + CNN to pass down the positioning features of the high-level of the infrared image pyramid and supplement the features of the low-level, so that the significant target features can be obtained, which is conducive to the category and position detection of small targets.

[0138] S334, digital mask processing is performed on the original infrared image according to the characteristics and position of the target object to obtain mask data, and the contour of the target object is obtained according to the mask data. The intersection position of 0 and 1 in the mask data is the position of the contour pixel point.

[0139] Figure 6 This is a flow chart of the fusion of dual infrared images and visible light images in the image feature fusion method proposed in the embodiment of this application. Figure 6 As shown, the following steps are included:

[0140] S341, determine the outline and the first position of the target object in the first image.

[0141] S342, mapping the contour of the target object to the visible light image according to the first projective transformation relationship and the first position to obtain a second position. The first projective transformation relationship is obtained by a registration algorithm, and the registration algorithm can refer to part or all of the implementation methods of steps S321-S329, which will not be repeated here.

[0142] S343, displaying the outline of the target object at a second position on the visible light image.

[0143] Exemplarily, the contour pixel points corresponding to the second position in the visible light image may be brightened to improve the recognition degree of the content captured by the lens.

[0144] S344, determining a third position of the target object in the second image, where the third position includes coordinates of pixel points of a contour of the target object in the second image.

[0145] S345, performing pixel-level fusion calculation on the pixel points in the third position of the second image and the pixel points in the second position of the visible light image to obtain a new pixel value; the third position and the second position conform to the second projective transformation relationship. The second projective transformation relationship is obtained by a registration algorithm, and the registration algorithm can refer to part or all of the implementation methods of steps S321-S329, which will not be repeated here.

[0146] S346, displaying the new pixel value of the pixel point within the outline of the target object on the visible light image.

[0147] Example 1

[0148] The image feature fusion method proposed in Example 1 of the present application can present the complete outline of the target object on the visible light image in the scene of reflection of street lights at night or strong reflection of opposite objects.

[0149] Figure 7 This is a scene diagram of the image feature fusion method proposed in Example 1 of the present application. It can be understood that the method can be executed by any model, device, platform, or device cluster with computing and processing capabilities. Figure 7 As shown, Example 1 is applied to a visual recognition system of a vehicle machine device, and the machine vision system includes a visible light lens and an infrared lens, the centers of the visible light lens and the infrared lens are on a horizontal plane, and the distance between the visible light lens and the infrared lens is less than 5 cm, wherein the pixel (resolution) of the visible light lens is greater than the pixel (resolution) of the infrared lens, and the focal lengths of the two lenses are the same, so that the pixel values ​​are aligned one by one. Based on the visual recognition system of the vehicle machine device, the image fusion method includes:

[0150] S401, the visible light lens and the infrared lens capture images of the same target object to obtain a first image and a visible light image; the first image is an image in a first video captured by the infrared lens, and the visible light image is an image in a visible light video captured by the visible light lens. The pixels (resolution) of the visible light video are greater than the pixels (resolution) of the infrared video, and the focal lengths are the same.

[0151] S402, obtaining a first projective transformation relationship according to registration of the first image and the visible light image.

[0152] S403, extracting the outline and the first position of the target object from the first image, where the first position is the coordinates of the pixels of the outline of the target object on the first image.

[0153] S404, performing feature-level fusion on the first image and the visible light image, taking the visible light image as a background, and determining a contour of the target object at a second position on the visible light image, wherein the second position and the first position conform to a first projective transformation relationship.

[0154] S405, highlighting the outline of the target object on the display screen of the vehicle device.

[0155] The image feature fusion method proposed in Example 1 of the present application can improve the vehicle equipment's ability to recognize people, vehicles, and objects on the road in scenes with insufficient light, such as late at night, in light fog, or in rainy weather, thereby improving the operational safety of the system.

[0156] Example 2

[0157] The image feature fusion method proposed in Example 2 of the present application improves the visual comfort of the image seen by the human eye and eliminates the "ghost eye" effect through feature-level fusion and pixel-level fusion of dual infrared images and visible light images in scenes where visible light is not clear.

[0158] The image feature fusion method proposed in Example 2 of the present application is applied to the visual recognition system of the vehicle equipment, and the machine vision system includes a visible light lens and two infrared lenses. The visible light lens can be set on the roof of the autonomous driving vehicle, and the two infrared lenses can be set at the positions of the headlights on both sides of the front of the vehicle. The distance between the infrared lenses is greater than 40cm to calculate the parallax of the binocular perspective, ensure the accuracy of the distance measurement, and improve the accuracy of the machine decision-making in the autonomous driving state.

[0159] Based on the visual recognition system of the vehicle equipment provided in Example 2, the image fusion method includes:

[0160] S501, a visible light lens, a first infrared lens, and a second infrared lens capture images of the same target object to obtain a first image, a second image, and a visible light image; the first image is an image in a first video captured by the first infrared lens, the second image is an image in a second video captured by the second infrared lens, and the visible light image is an image in a visible light video captured by the visible light lens. The pixels (resolution) of the visible light video are greater than the pixels (resolution) of the infrared video, and the focal lengths are the same.

[0161] S502, performing deep fusion on the first image, the second image and the visible light image to obtain a fused image, wherein the deep fusion includes feature-level fusion and pixel-level fusion, and the specific steps may refer to some or all of the implementations of S341-S346, which will not be described in detail here.

[0162] S503, displaying the fused image of the target object on the display screen of the vehicle equipment.

[0163] The image fusion method provided in Example 2 of the present application can also be applied to character identification. For identification based on key information such as character facial information, behavioral habit information (such as gait recognition), and skeleton, the recognition ability of the visual system is enhanced by accurately fusing depth information and image detail information.

[0164] Example 3

[0165] The image feature fusion method proposed in Example 3 of the present application can be used to measure the distance of a target object. In Example 3, a machine vision system as in Example 2 is used, including a visible light lens and two infrared lenses.

[0166] Figure 8 This is a schematic diagram of the image feature fusion method proposed in Example 3 of the present application for measuring the distance of a target object. Figure 3 As shown, the following steps are included:

[0167] S601, performing deep fusion on the first image, the second image and the visible light image to obtain a fused image, wherein the deep fusion includes feature-level fusion and pixel-level fusion, and the specific steps may refer to part or all of the implementations of S341-S346, which will not be described in detail here.

[0168] S602: Determine the characteristics of the target object according to the fused image.

[0169] S603, performing disparity calculation based on the first image and the second image, and obtaining a disparity grayscale image based on difference encoding. The disparity is the difference in x-coordinates corresponding to the same space point in two infrared images.

[0170] S604: Calculate the target object distance according to the disparity grayscale image and the target object features.

[0171] S605, indicating the distance of the target object on the display screen of the vehicle device through the brightness of the grayscale of the target object image, and the closer the grayscale image is to the camera, the brighter it is.

[0172] In one scenario, the machine vision system includes a laser detection device; the laser detection device transmits / receives laser signals, and the machine vision system fuses the intensity image of the returned laser signal reflected by the target object with the visible light image, and displays the target object and the corresponding real target distance on the visual image.

[0173] In one scenario, the machine vision system includes a millimeter-wave radar device; the millimeter-wave radar device transmits / receives millimeter-wave signals, and the machine vision system fuses the echo signal reflected by the target object with the visible light image to obtain the corresponding real target distance.

[0174] The image feature fusion method proposed in the embodiment of the present application uses the visible light image as the background, so that the contours of the infrared image and the visible light image are accurately fused, thereby reducing the discomfort caused to the user by the synthesized image and enhancing the user experience.

[0175] An image feature fusion method proposed in an embodiment of the present application is applied to a vehicle-computer recognition system. It can improve the vehicle-computer system's ability to recognize pedestrians on the road and improve the operational safety of the system in scenes with insufficient light, such as late at night, in fog, or in rainy weather.

[0176] An image feature fusion method proposed in an embodiment of the present application is applied to a human action recognition scenario to identify changes in people, vehicles, and objects within the camera field of view in dark or other low-light scenes, thereby improving recognition accuracy.

[0177] An image feature fusion method proposed in an embodiment of the present application is applied to a character identity recognition scenario. The recognition capability based on key information such as character facial information and behavioral habit information (such as gait recognition) is enhanced by accurately fusing distance information and image detail information.

[0178] The application scenarios of the image feature fusion method proposed in the embodiments of the present application include but are not limited to the scenarios provided in the above embodiments.

[0179] An embodiment of the present application provides an electronic device, comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute a method such as any one of the image feature fusion methods.

[0180] An embodiment of the present application provides a machine vision system, which includes a first lens, a second lens, a visible light lens, and a device for image feature level fusion; wherein the resolution of the first lens is lower than that of the visible light lens; the second lens has the same resolution as the first lens; the first lens is used to capture a first video; the second lens is used to capture a second video; the visible light lens is used to capture visible light video; and the device for image feature level fusion is used to execute a method such as any one of the image feature fusion methods.

[0181] In a possible implementation, the machine vision system further includes a laser detection device; the image feature level fusion device fuses the laser signal returned by the laser detection device to obtain the distance of the real target corresponding to the outline of the target object on the visible light image.

[0182] In a possible implementation, the machine vision system further includes a millimeter wave radar device; the image feature level fusion device fuses the millimeter wave signal returned by the millimeter wave radar device to obtain the distance of the real target corresponding to the outline of the target object on the visible light image.

[0183] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0184] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0185] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0186] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

1. An image feature fusion method, characterized in that: Applied in a machine vision system, the machine vision system comprises at least a first lens and a visible light lens; the resolution of the first lens is lower than that of the visible light lens, and the method comprises: Capturing multiple mode images of the same object; the multiple mode images include a first image and a visible light image; the first image is an image in a first video captured by a first lens, and the visible light image is an image in a visible light video captured by the visible light lens; Obtaining a first projective transformation relationship according to registration of the first image and the visible light image; Extracting features and a first position of a target object from the first image, wherein the features of the target object include a contour of the target object, and the first position includes coordinates of pixel points of the contour of the target object on the first image; The first image and the visible light image are fused at a feature level, and a contour of the target object is displayed at a second position on the visible light image, and the second position and the first position conform to the first projective transformation relationship.

2. The method according to claim 1, characterized in that The machine vision system further includes a second lens, the second lens has the same resolution as the first lens, the multiple mode images further include a second image captured by the second lens, and the method further includes: Obtaining a second projective transformation relationship according to registration of the second image and the visible light image; Extracting the features and a third position of the target object from the second image, wherein the third position includes the coordinates of the pixel points of the outline of the target object on the second image; Performing pixel-level fusion calculation on a pixel point in a third position of the second image and a pixel point in a second position on the visible light image to obtain a new pixel value; the third position and the second position conform to the second projective transformation relationship; The new pixel value is used as the pixel value of the pixel point within the outline of the target object on the visible light image.

3. The method according to claim 1 or 2, characterized in that: The obtaining a first projective transformation relationship according to the registration of the first image and the visible light image includes: Collecting a plurality of sets of image pairs of the first image and the visible light image for the same object; The first projective transformation relationship is obtained by calculating the coordinates of a plurality of points on the plurality of groups of image pairs of the first images and the visible light images.

4. The method according to any one of claims 1 to 4, characterized in that: The step of extracting the feature and the first position of the target object from the first image includes: The first image is input into a neural network recognition model, and the contour features and the first position of the target object are obtained according to the output of the neural network recognition model; wherein the neural network recognition model includes a FYOLACT network.

5. The method according to claim 4, characterized in that The step of obtaining the contour feature and the first position of the target object according to the output of the neural network recognition model includes: The FYOLACT network divides the first image into images of different sizes to obtain an image pyramid; the bottom layer of the image pyramid is a high-resolution representation of the first image, and the top layer of the image pyramid is a low-resolution representation of the first image. The lower the resolution, the smaller the corresponding image and the higher the level. Perform convolution and differential calculation on each layer of the image pyramid through a convolutional network model, and output feature distribution of each layer of the image; According to the feature distribution of each layer of the image, the feature pyramid network is used in combination with the feedforward calculation of CNN to predict and output the contour features of the target object and the coordinates of the corresponding pixel points.

6. The method according to any one of claims 1 to 5, characterized in that: The features also include one or a combination of the following: category features, morphological features, motion features and identity features of the target object, and the first position also includes one or a combination of the following: coordinates of pixel points corresponding to the category features, morphological features, motion features and identity features on the first image.

7. The method according to any one of claims 2 to 6, characterized in that: The method further comprises: The first image and / or the second image is subjected to image quality enhancement processing, wherein the image quality enhancement processing includes video noise reduction, deblurring, sharpening and / or de-shaking.

8. The method according to any one of claims 2 to 6, characterized in that: The method further comprises: Perform video super-resolution processing on the first image and / or the second image.

9. The method according to any one of claims 2 to 8, characterized in that: The method further comprises: Calculate the disparity of the first image and the second image, and obtain a disparity grayscale image according to the difference encoding; The distance of the target object is obtained based on the disparity grayscale image and the characteristics of the target object.

10. The method according to any one of claims 1 to 9, characterized in that: The first lens and the second lens are infrared lenses.

11. The method according to any one of claims 1 to 10, characterized in that: The machine vision system includes a laser detection device; the laser detection device transmits / receives a laser signal, and the method includes: The real target distance of the target object is obtained by fusing the returned laser signal reflected by the target object with the visible light image.

12. The method according to any one of claims 1 to 11, characterized in that: The machine vision system includes a millimeter wave radar device; the millimeter wave radar device transmits / receives a millimeter wave radar signal, and the method includes: The real distance of the target object is obtained by fusing the returned millimeter-wave radar signal reflected by the target object with the visible light image.

13. A device for image feature level fusion, characterized in that: Applied in a machine vision system, the machine vision system at least comprises a first lens and a visible light lens; the resolution of the first lens is lower than that of the visible light lens, and the device comprises: An image acquisition module, used to acquire multiple mode images of the same object; the multiple mode images include a first image and a visible light image; the first image is an image in a first video acquired by a first lens, and the visible light image is an image in a visible light video acquired by the visible light lens; A first image processing module is used to obtain a first projective transformation relationship according to the registration of the first image and the visible light image; extract features and a first position of a target object from the first image, wherein the features of the target object include a contour of the target object, and the first position includes coordinates of pixel points of the contour of the target object on the first image; A first image fusion module, used for performing feature-level fusion on the first image and the visible light image; The first display module is used to display the outline of the target object at a second position on the visible light image, and the second position and the first position conform to the first projective transformation relationship.

14. The device according to claim 13, characterized in that The machine vision system further includes a second lens, the second lens has the same resolution as the first lens, the multiple mode images also include a second image captured by the second lens, and the device further includes: A second image processing module is used to obtain a second projective transformation relationship according to the registration of the second image and the visible light image; extract the features and a third position of the target object from the second image, wherein the third position includes the coordinates of the pixel points of the outline of the target object on the second image; a second image fusion module, configured to perform pixel fusion calculation on a pixel point in a third position of the second image and a pixel point in a second position on the visible light image to obtain a new pixel value; the third position and the second position conform to the second projective transformation relationship; The second display module is used to display the new pixel value as the pixel value of the pixel point within the outline of the target object on the visible light image.

15. An electronic device, characterized in that: include: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 12.

16. A machine vision system, characterized in that: The machine vision system comprises a first lens, a second lens, a visible light lens and an image feature level fusion device; wherein the resolution of the first lens is lower than that of the visible light lens; and the resolution of the second lens is the same as that of the first lens; The first lens is used to capture a first video; The second lens is used to capture a second video; The visible light lens is used to collect visible light video; The device for image feature level fusion is used to execute the method according to any one of claims 1 to 12 to obtain a fused image.

17. The system according to claim 16, characterized in that The machine vision system also includes a laser detection device; Laser detection equipment transmits / receives laser signals; The image feature level fusion device fuses the laser signal reflected by the target object with the visible light image.

18. The system according to claim 16, characterized in that The machine vision system also includes a millimeter wave radar device; The millimeter wave radar device transmits / receives millimeter wave signals; The image feature level fusion device fuses the millimeter wave signal reflected by the target object with the visible light image.

Images