Method and System for Fusion of Visual Inspection Perspective in Mixers

Abstract

This invention relates to the field of image processing and discloses a method and system for fusion of viewing views in a mixer. The method includes: acquiring multiple images of the mixer, wherein at least two of the multiple images include overlapping regions; extracting feature point groups from the multiple images, wherein the feature point groups include associated feature points located in the overlapping regions of the at least two images; selecting matching feature point groups from the feature point groups based on the similarity between the feature points in the feature point groups; determining a transformation matrix based on the matching feature point groups; and using the transformation matrix to transform the multiple images to the same plane.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular, to a method and system for fusing the view field of a mixer. BACKGROUND

[0002] In the production process of concrete, in order to observe the mixing situation of concrete mixture in real time, the manufacturer usually installs a viewing device on the upper cover of the mixer. The existing viewing device is usually composed of a camera and a butterfly valve. When the feeding of each raw material of concrete is completed, the butterfly valve will automatically open, and the camera will shoot the real-time mixing picture, and the image will be transmitted back to the control room. The staff in the control room can thus master the mixing situation of concrete in real time and complete the concrete operation in time, avoiding the situation that the uniformity of concrete does not meet the standard due to too short mixing time, affecting the quality of the finished product, or affecting the production efficiency due to too long mixing time.

[0003] At present, the mixer usually uses a single camera to collect mixing images, and the collection image view field usually only includes a small part of the internal area of the mixer, making it difficult for the staff to view the entire internal area of the mixer. Therefore, when mixing concrete, the staff can only observe the uniformity of the concrete in the local area, and when cleaning the mixer, they can also only observe the local area, which may cause incomplete cleaning of the mixer, leaving concrete inside, and the local view field of the viewing device may easily cause the staff to overlook the problem of bearing shaft, resulting in too much concrete adhering to the mixing shaft, thereby affecting the production efficiency. SUMMARY

[0004] The purpose of the present application is to overcome the problem that the viewing field of the existing mixer viewing device cannot cover the entire internal area of the mixer, and to provide a method and system for fusing the viewing field of a mixer.

[0005] To achieve the above-mentioned purpose, the first aspect of the present application provides a method for fusing the viewing field of a mixer, comprising:

[0006] obtaining a plurality of images of the internal area of the mixer, wherein at least two images of the plurality of images include an overlapping area;

[0007] extracting a feature point group from the plurality of images, the feature point group including associated feature points located in the overlapping area of the at least two images, respectively;

[0008] selecting a matching feature point group from the feature point group according to the similarity between the feature points in the feature point group;

[0009] determining a transformation matrix according to the matching feature point group;

[0010] transforming the plurality of images to the same plane using the transformation matrix.

[0011] In an embodiment of the present application, the method further comprises:

[0012] According to the preset fusion degree, linearly fusing the images transformed to the same plane.

[0013] In an embodiment of the present application, the method further comprises:

[0014] Preprocessing the multiple images to remove image noise and image distortion.

[0015] In an embodiment of the present application, the matching feature point groups are selected from the feature point groups according to the similarity between the feature points included in the feature point groups, comprising:

[0016] Determining the Euclidean distance between the feature points included in the feature point groups, and determining the similarity according to the Euclidean distance;

[0017] Selecting the feature point groups with the similarity reaching a first threshold value as the matching feature point groups.

[0018] In an embodiment of the present application, the transformation matrix is determined according to the matching feature point groups:

[0019] Determining multiple matching feature point systems, wherein each matching feature point system includes multiple groups of matching feature point groups randomly selected;

[0020] Respectively determining the pre-selected transformation matrices corresponding to the multiple matching feature point systems;

[0021] Determining the distance values between the matching feature points included in the matching feature point groups transformed by each pre-selected transformation matrix;

[0022] According to the distance values, judging whether the transformed matching feature point groups are inner point groups, wherein the inner point groups are the transformed matching feature point groups with the distance values less than a second threshold value;

[0023] Selecting the pre-selected transformation matrix with the most inner point groups as the transformation matrix.

[0024] In an embodiment of the present application, the feature point groups in the multiple images are extracted, comprising:

[0025] Selecting a feature extractor with scale invariance and illumination invariance to extract the feature point groups.

[0026] In an embodiment of the present application, the multiple images are transformed to the same plane by using the transformation matrix, comprising:

[0027] Transforming the matching feature point groups by using the transformation matrix to align the matching feature points included in the matching feature point groups.

[0028] The second aspect of the present application provides a processor configured to execute the above-mentioned look-see field-of-view fusion method for a mixer.

[0029] The third aspect of the present application provides a look-see field-of-view fusion system for a mixer, comprising:

[0030] an image acquisition module comprising at least two cameras for acquiring images of the interior of the mixer; and

[0031] the above-mentioned processor.

[0032] The fourth aspect of the present application provides a machine-readable storage medium, characterized in that the machine-readable storage medium has instructions stored thereon, which, when executed by a processor, cause the processor to implement the above-mentioned look-see field-of-view fusion method for a mixer.

[0033] Through the above technical solution, multiple cameras are used to acquire images of multiple regions inside the mixer, and the multiple images are spliced and fused to form a complete observation field of view inside the mixer. Compared with the field of view of the traditional single camera look-see device, the visual area covered by the technical solution is wider, overcoming the limitation of the local observation field of view, facilitating the operator to observe the real-time working condition inside the host machine (the real-time working condition mainly includes the uniformity state of the mixed concrete, whether the host machine is completely cleaned, and the shaft holding working condition of the mixing shaft), and providing protection for the quality of the concrete and the maintenance of the host machine.

[0034] Other features and advantages of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, and are used together with the following specific embodiments to explain the present application, but do not constitute a limitation on the present application. In the drawings:

[0036] Figure 1 schematically shows a flowchart of a look-see field-of-view fusion method for a mixer according to an embodiment of the present application;

[0037] Figure 2 schematically shows a flowchart of determining a transformation matrix according to a matching feature point group according to an embodiment of the present application; and

[0038] Figure 3 schematically shows a structural diagram of a look-see field-of-view fusion system for a mixer according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended for explanation and interpretation of the present application, and are not intended to limit the present application.

[0040] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship, motion condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications will also change accordingly.

[0041] In addition, if the embodiments of the present application involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the fact that a person skilled in the art can realize it, and when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist, and is not within the protection scope claimed by the present application.

[0042] Figure 1 The flowchart of the field of view fusion method for the blender according to the embodiments of the present application is schematically shown as follows, Figure 1 In one embodiment of the present application, a field of view fusion method for a blender is provided, which can include the following steps:

[0043] Step S101: acquiring a plurality of images for the inside of the blender;

[0044] The processor expands and fuses the inside view of the blender based on the images collected by the built-in camera inside the blender. In order to effectively expand the inside view, the number of built-in cameras should be more than two to collect images of different areas inside the blender, so the number of images acquired by the processor is also two or more. The image fusion process performed by the processor is the process of splicing and fusing the acquired images, wherein the splicing of images is to transform the images collected under different perspectives, illuminations and scales to the same plane to form a complete, expanded and global view under the same perspective.

[0045] In an embodiment of the present application, when the processor performs the image stitching, the processor first stitches two images, and then stitches the stitched image and the next image, until all the images are stitched. For example, the processor acquires three images of the inside of the blender, image1, image2 and image3. The processor first stitches image1 and image2 to obtain a stitched image image1S, and then stitches image1S and image3 to obtain a stitched image image2S. Therefore, when acquiring the images, the processor needs to ensure that at least two of the images include an overlapping region, which refers to the same part of the inside of the blender in different images. The image stitching relies on the feature points in the overlapping region.

[0046] In an embodiment of the present application, the processor pre-processes the acquired images to remove image noise and image distortion, so that the subsequent image stitching is more accurate. Further, a Gaussian filter function can be used to remove image noise, a histogram can be used to equalize the image, and image distortion can be corrected.

[0047] In an embodiment of the present application, as shown in FIG. 1, Figure 1 The field of view fusion method provided by the present application further includes:

[0048] Step S102: Extract a feature point group from the plurality of images, the feature point group including associated feature points respectively located in overlapping regions of at least two images.

[0049] After the processor acquires the plurality of images and pre-processes the images, the processor extracts associated feature points in the images to form a feature point group, the feature point group including associated feature points respectively located in overlapping regions of at least two images. For example, if the processor acquires two images, one feature point group includes two feature points, the two feature points are respectively located in overlapping regions of the two images, and the two feature points are associated points in the two images. The degree of association between the two feature points is positively correlated with the similarity of the two feature points.

[0050] In an embodiment of the present application, a feature extractor with scale invariance and illumination invariance is selected to extract the feature point group from the plurality of images. The feature extractor first finds points with significant association in the plurality of images, then describes the points respectively, and finally compares the degree of similarity of the descriptions to determine whether the points are feature points with association. If the scale can be determined before the feature description, the scale invariance can be achieved. The principle of scale invariance is to transform all the images to the same scale before describing a point, and then describe the point on the unified standard. Similarly, the illumination invariance of the feature extractor means that the feature extractor can still well recognize and describe the feature points under different illumination conditions.

[0051] In an embodiment of the present application, SIFT algorithm is used as a feature extractor to extract the feature point group. The essence of SIFT algorithm is to find the significant points in different scale spaces, and each point is mathematically described as a 128-dimensional vector {q1,..., q 128 The points found by SIFT are some very prominent points, which do not change due to factors such as illumination, affine transformation and noise, such as corner points, edge points, bright points in dark areas and dark points in bright areas.

[0052] In an embodiment of the present application, as shown in Figure 1 The field of view fusion method provided by the present application further comprises:

[0053] Step S103: screening a matching feature point group from the feature point group according to the similarity between the feature points in the feature point group.

[0054] In an embodiment of the present application, the similarity between the feature points can be described by Euclidean distance, and the calculation formula of Euclidean distance is:

[0055]

[0056] Where (x i ,y i ) and (x′ i ,y′ i ) are the coordinates of the feature points contained in the feature point group.

[0057] The greater the Euclidean distance between the feature points, the smaller the similarity. In an embodiment of the present application, a numerical conversion relationship between the Euclidean distance between the feature points and the similarity can be established, which expresses the negative correlation and linear correlation between them. The processor presets a first threshold as a screening condition, and screens the feature point group whose similarity reaches the first threshold as the matching feature point group, which is used for subsequent calculation of the transformation matrix.

[0058] In an embodiment of the present application, as shown in Figure 1 The field of view fusion method provided by the present application further comprises:

[0059] Step S104: determining the transformation matrix according to the matching feature point group.

[0060] The purpose of image stitching is to convert images of different perspectives to the same perspective, i.e. to transform images of different perspectives to the same plane, and the transformation process includes alignment of matching feature points in the matching feature point group, and alignment between matching feature points, i.e. transformation of the matching feature point group by the transformation matrix, so as to map the matching feature points in one image to the corresponding matching feature points in another image by the transformation matrix.

[0061] Figure 2 A flowchart of determining a transformation matrix according to matched feature point groups is shown according to an embodiment of the present application, as shown in Figure 2 In one embodiment of the present application, step S104 comprises:

[0062] Step S401: Determine a plurality of matched feature point groups.

[0063] Step S402: Determine a preselected transformation matrix corresponding to each matched feature point group, respectively.

[0064] Step S403: Determine the distance value between matched feature points included in the matched feature point group transformed by each preselected transformation matrix.

[0065] Step S404: Determine whether the transformed matched feature point group is an inlier group according to the distance value.

[0066] Step S405: Select the preselected transformation matrix with the most inlier groups as the transformation matrix.

[0067] In one embodiment of the present application, a matched feature point group containing a plurality of matched feature point groups is determined, and the specific number of matched feature point groups included in each matched feature point group can be determined by the total number of matched feature point groups obtained, which is not limited in the present application.

[0068] In one embodiment of the present application, each matched feature point group includes four matched feature point groups, and the transformation matrix H corresponding to each matched feature point group is calculated as a preselected transformation matrix using formula (1) and formula (2):

[0069]

[0070]

[0071] Wherein, (x1, y1) and (x'1, y'1), (x2, y2) and (x'2, y'2), (x3, y3) and (x'3, y'3), (x4, y4) and (x'4, y'4) are the coordinates of the matched feature points contained in the four matched feature point groups in the two images.

[0072] In one embodiment of the present application, after the processor calculates and obtains a plurality of preselected transformation matrices H, all matched feature point groups are transformed by the preselected transformation matrices, the distance value between matched feature points included in the matched feature point group transformed by each preselected transformation matrix is determined, and the distance value t between each matched feature point group can be calculated by formula (3):

[0073]

[0074] wherein (x i ,y i ) and (x′ i ,y′ i ) are coordinates of the matched feature points contained in the matched feature point group, and H is the pre-selected transformation matrix.

[0075] In an embodiment of the present application, the processor pre-sets a second threshold as an inlier group screening condition, and the feature point group with a distance value t less than the second threshold after being changed by the pre-selected transformation matrix H is an inlier group, so that the pre-selected transformation matrix from which the inlier group is obtained after the matched feature point group is changed is the pre-selected transformation matrix with the best alignment effect, and the pre-selected transformation matrix is determined as the transformation matrix for image stitching.

[0076] In an embodiment of the present application, as shown in Figure 1 , the field of view fusion method provided by the present application further comprises:

[0077] Step S105: transforming the plurality of images to the same plane using the transformation matrix.

[0078] Transforming the images to the same plane means transforming the matched feature point group using the transformation matrix, and the transformation process is the alignment of the matched feature points in the matched feature point group. The alignment between the matched feature points is achieved by mapping the matched feature points in one image to the corresponding matched feature points in another image through the transformation matrix.

[0079] In an embodiment of the present application, the field of view fusion method provided by the present application further comprises:

[0080] Linearly fusing the images transformed to the same plane according to a pre-set fusion degree.

[0081] After the plurality of images are transformed to the same plane (after the stitching is completed), due to the difference in light intensity between the images, the existence of the difference in viewing angle, and the false matching of the feature extractor when extracting the feature point group, false images may appear in the stitched image. In an embodiment of the present application, the false images are eliminated or reduced through image linear fusion, so that the joint at the stitching position is more natural. Image linear fusion refers to linearly adding the pixel values at the same positions of the input two images f(x) and h(x), and then assigning the result to the pixel at the same position of the fused image.

[0082] Image linear fusion can be completed by formula (4):

[0083] g(x)=a×f(x)+(1-a)×h(x) (4)

[0084] Wherein, g(x) is the fused image, f(x) and h(x) are the images to be fused, and a is the fusion degree of the image. The fusion degree a controls the weight of the two images to be fused in the fused image, and the value of a is 0-1. If a is gradually reduced from 1 to 0, the superposition effect when the image f(x) is transitioned to the image h(x) can be generated. In an embodiment of the present application, the specific value of the fusion degree a can be determined by the staff during the on-site debugging of the viewing field fusion system.

[0085] In an embodiment of the present application, a processor is provided for executing the viewing field fusion method in the above-mentioned embodiments.

[0086] Figure 3 The structure of a viewing field fusion system for a mixer according to an embodiment of the present application is schematically shown in FIG. 1. As shown in FIG. 1, in an embodiment of the present application, a viewing field fusion system for a mixer is provided, which comprises: Figure 3 As shown in FIG. 1, in an embodiment of the present application, a viewing field fusion system for a mixer is provided, which comprises:

[0087] The processor 3 in the above-mentioned embodiments; and

[0088] An image acquisition module 1 comprising at least two cameras for acquiring images of the inside of the mixer 2, each camera capturing images of a different area inside the mixer 2, the image acquisition module 1 being in communication connection with the processor 3, and the image acquisition module 1 transmitting the images of different areas and different angles inside the mixer 2 acquired to the processor 3 for image splicing and fusion.

[0089] In an embodiment of the present application, the viewing field fusion system further comprises a display module 4 for displaying the images obtained by the processor 3 and the images after splicing and fusion by the processor 3.

[0090] When the operator is working, the operator can observe the uniformity state of the concrete inside the mixer 2 globally through the display module 4, and can complete the concrete discharging operation according to the uniformity state of the concrete, thereby avoiding the situation that the uniformity of the concrete is not up to the standard due to the too short mixing time, which affects the quality of the finished product of the concrete, or the situation that the production efficiency is affected due to the too long mixing time. The operator can also check the cleaning situation inside the mixer 2 through the display module 4 when the mixer 2 is cleaned, thereby avoiding the situation that the concrete is left due to incomplete cleaning and the situation that the shaft is stuck due to the concrete clumps adhered to the shaft.

[0091] In an embodiment of the present application, a machine readable storage medium is provided, and the machine readable storage medium stores instructions which, when executed by the processor 3, cause the processor 3 to implement the viewing field fusion method in the above-mentioned embodiments.

[0092] In one embodiment of the present application, a computer program product is also provided, comprising a computer program which, when executed by a processor, implements the look field fusion method for a blender according to the above-mentioned embodiments.

[0093] Those skilled in the art will appreciate that embodiments of the present application can be supplied as a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0094] The present application is described in reference to the flowchart illustrations and / or block diagrams according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart illustrations and / or block diagrams, and combinations of flows and / or blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0095] In one typical arrangement, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0096] ​​​Memory can include non-persistent memory, Random Access Memory (RAM), and / or non-volatile memory, such as read only memory (ROM) or flash memory, in a computer readable medium. Memory is an example of computer readable media.

[0097] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0098] It should also be noted that the terms "comprising", "containing", or any other variant thereof, are intended to encompass a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements in the list, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0099] The above merely provides an example of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for blending machine's view fusion, characterized in that, The method comprises: acquiring a plurality of images of the inside of the mixer, wherein at least two of the plurality of images comprise an overlapping region; extracting a feature point group from the plurality of images, the feature point group comprising associated feature points respectively located in the overlapping region of the at least two images; selecting a matching feature point group from the feature point group according to the similarity between the feature points in the feature point group; determining a transformation matrix according to the matching feature point group; transforming the plurality of images to the same plane using the transformation matrix; the determining of the transformation matrix according to the matching feature point group comprises: determining a plurality of matching feature point systems, wherein each matching feature point system comprises a plurality of groups of randomly selected matching feature point groups; determining a preselected transformation matrix corresponding to each matching feature point system respectively; determining the distance value between the matching feature points included in the matching feature point group transformed by each preselected transformation matrix; judging whether the transformed matching feature point group is an inlier group according to the distance value, wherein the inlier group is the transformed matching feature point group with a distance value less than a second threshold value; selecting the preselected transformation matrix with the largest number of inlier groups as the transformation matrix.

2. The surveillance see-through fusion method of claim 1, wherein, The method further comprises: linearly fusing the images transformed to the same plane according to a preset fusion degree.

3. The surveillance see-through fusion method of claim 1, wherein, The method further comprises: preprocessing the plurality of images to remove image noise and image distortion.

4. The surveillance see-through fusion method of claim 1, wherein, The selecting of the matching feature point group according to the similarity between the feature points in the feature point group comprises: determining the Euclidean distance between the feature points in the feature point group, and determining the similarity according to the Euclidean distance; selecting the feature point group with a similarity reaching a first threshold value as the matching feature point group.

5. The surveillance see-through fusion method of claim 1, wherein, The extracting of the feature point group from the plurality of images comprises: selecting a feature extractor with scale invariance and illumination invariance to extract the feature point group.

6. The surveillance see-through fusion method of claim 1, wherein, The transforming of the plurality of images to the same plane using the transformation matrix comprises: transforming the matching feature point group using the transformation matrix to align the matching feature points included in the matching feature point group.

7. A see what you're stirring system for a blender, comprising: The method comprises: an image acquisition module comprising at least two cameras for acquiring images of the inside of the mixer; and a processor configured to execute the method for fusing the view field of the mixer according to any one of claims 1 to 6. The machine readable storage medium stores instructions which, when executed by the processor, cause the processor to implement the method for fusing the view field of the mixer according to any one of claims 1 to 6.

8. A machine-readable storage medium, characterized in that, ​

Images