Image processing method, apparatus and system based on artificial intelligence, and storage medium

Abstract

The present application discloses an image processing method, apparatus and system based on artificial intelligence, and a storage medium. The method comprises: acquiring a calibration image of a calibration apparatus, wherein the calibration apparatus comprises a pattern plate with a plurality of patterns; performing feature extraction on the calibration image to obtain a first feature map; inputting the first feature map into a frame regression network to output a plurality of regions of interest corresponding to the calibration apparatus in the first feature map; generating a preset number of candidate frames for each region of interest on the basis of a preset prior frame, the size of each candidate frame being the same as that of the preset prior frame, and the preset number being determined on the basis of the number of the patterns; and inputting all the candidate frames and the first feature map into a center regression network to obtain image center point coordinates of each pattern in the calibrated image. The solution can obtain clear imaging edge information, even if the imaging quality is poor, an accurate candidate region can be obtained, and the detection precision of image recognition is significantly improved.

Description

Artificial intelligence-based image processing method, device, system and storage medium

[0001] Cross-reference to Related Applications

[0002] This application claims priority to Chinese Patent Application No. 202410531294.4, filed on April 29, 2024, the contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of image processing technology, in particular to an artificial intelligence-based image processing method, device, system and storage medium. BACKGROUND

[0004] With the development of intelligent technology, image acquisition devices are widely used in various fields. For example, visible light cameras, infrared thermal imaging cameras, and dual-light cameras composed of visible light cameras and infrared thermal imaging cameras all need to calibrate the camera's intrinsic parameters to correct the distortion due to different degrees of lens distortion. For example, for the application of fire source positioning to achieve automatic fire extinguishing, the camera's intrinsic parameters must be accurately calibrated. However, accurate calibration of camera intrinsic parameters requires high resolution of camera imaging. For example, infrared cameras sense the distribution of external infrared radiation energy to form an image, resulting in less clear imaging than visible light imaging. Mainly in the form of low image resolution, blurred object edge contour, and high image noise, especially large-area temperature anomalies that can cause image feature extraction to fail. Therefore, it also causes the problem of large errors in extracting infrared image features using common feature extraction algorithms. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide an artificial intelligence-based image processing method, device, system and storage medium to at least partially solve the above technical problems.

[0006] To achieve the above-mentioned purpose, the first aspect of the present application provides an artificial intelligence-based image processing method, comprising: collecting a calibration image of a calibration device, the calibration device comprising a pattern board having a plurality of patterns; performing feature extraction on the calibration image to obtain a first feature map after processing; inputting the first feature map into a bounding box regression network to output a plurality of regions of interest corresponding to the calibration device in the first feature map through the bounding box regression network; generating a preset number of candidate boxes for each region of interest based on a preset prior box, the size of each candidate box being the same as the size of the preset prior box, and the preset number being determined according to the number of the plurality of patterns; inputting all candidate boxes and the first feature map into a center regression network to obtain the coordinates of the image center point of each pattern in the calibration image.

[0007] In the embodiments of the present application, inputting all the candidate boxes into the center regression network to determine the image center point of each pattern in the calibration image according to the predicted image coordinates output by the center regression network comprises: inputting the first feature map and all the candidate boxes into the center regression network to output a plurality of predicted image coordinates and a predicted probability corresponding to each preset image coordinate according to the center regression network; arranging the plurality of predicted probabilities from high to low, and determining the predicted image coordinates corresponding to the preset number of predicted probabilities arranged in the front as the first image coordinates of the image center point of each pattern in the calibration image.

[0008] In the embodiments of the present application, the plurality of patterns are a plurality of same and regularly arranged patterns, and generating a preset number of candidate boxes for each region of interest based on a preset anchor box comprises: obtaining the size of the circumscribed rectangle of any one pattern; generating a preset anchor box according to the size of the circumscribed rectangle; inputting the preset anchor box and the region of interest into the region generation module to output a preset number of candidate boxes for each region of interest through the region generation module.

[0009] In the embodiments of the present application, the function expression of the region generation module is as shown in the following formula (1):

[0010] Wherein, Proposal (k,n) refers to the nth candidate box of the kth region of interest, bbox k refers to the kth region of interest, Base_anchor n refers to the nth candidate box generated based on the preset anchor box, x1, x2 respectively refer to the horizontal coordinates of the diagonal points of the circumscribed rectangle corresponding to the preset anchor box, y1, y2 respectively refer to the vertical coordinates of the diagonal points of the circumscribed rectangle corresponding to the preset anchor box, x, y refer to the horizontal coordinates and vertical coordinates of the fixed point of each region of interest, w, h respectively refer to the width and height of each region of interest.

[0011] In the embodiments of the present application, processing the calibration image to obtain the processed first feature map comprises: inputting the calibration image into the residual network and outputting a plurality of second feature maps with different scales through the residual network; inputting the plurality of second feature maps into the up-sampling module and outputting a plurality of third feature maps with the same size through the up-sampling module; connecting all the third feature maps to obtain a fourth feature map;

[0012] Inputting the fourth feature map into the attention mechanism module and outputting the first feature map after the spatial dimension and channel dimension of the fourth feature map are pooled by the attention mechanism module.

[0013] In embodiments of the present application, the method further comprises: sorting the actual center points of all the patterns on the pattern board, and obtaining the relative positions between all the sorted actual center points; determining the world coordinates of each sorted actual center point according to the relative position corresponding to each sorted actual center point; performing perspective transformation on the first image coordinates of all the image center points based on the relative positions of all the sorted actual center points, to determine the first image coordinates corresponding to each sorted actual center point; and determining the intrinsic parameters of the image acquisition device that acquires the calibration image according to all the first image coordinates and world coordinates based on the corresponding relationship between the first image coordinates and the world coordinates of each sorted actual center point.

[0014] In embodiments of the present application, the arrangement of the plurality of patterns is an array, the relative positions include the arrangement interval of the array, the number of rows of the array is a first quantity, the number of columns of the array is a second quantity, and sorting the actual center points of all the patterns includes: sorting the actual center points of all the patterns according to the order of first row and then column based on the array; performing perspective transformation on the first image coordinates of all the image center points based on the relative positions of all the sorted actual center points, to determine the first image coordinates corresponding to each sorted actual center point, includes: after determining the first image coordinates of each image center point, determining two opposite corners in all the image center points according to all the first image coordinates; determining a plurality of first corner points of a first target polygon constructed in the calibration image according to the two opposite corners; determining a plurality of second corner points corresponding to the shape of the pattern board according to the arrangement interval of the array, the first quantity and the second quantity, to construct a second target polygon corresponding to the shape of the pattern board in the calibration image through the plurality of second corner points; generating a second image coordinate corresponding to each sorted actual center point in the second target polygon based on the arrangement interval of the array; determining a perspective transformation matrix between the first target polygon and the second target rectangle according to the plurality of first corner points and the plurality of second corner points; converting each first image coordinate through the perspective transformation matrix to determine the second image coordinate corresponding to each first image coordinate, to determine the first image coordinates corresponding to each sorted actual center point.

[0015] In embodiments of the present application, determining a plurality of first corner points of a first target polygon constructed in the calibration image according to the two opposite corners includes: for any one of the two opposite corners, determining a connecting line between each center point and the opposite corner to determine the number of patterns passing through each connecting line; for each opposite corner, connecting lines with the number of passing patterns being a first quantity are first target connecting lines, and connecting lines with the number of passing patterns being a second quantity are second target connecting lines; determining connecting line intersection points between all the first target connecting lines and all the second target connecting lines; and determining all the opposite corners and all the connecting line intersection points as the plurality of first corner points of the first target polygon constructed in the calibration image.

[0016] The second aspect of the present application provides an image processing device based on artificial intelligence, comprising:

[0017] a memory configured to store instructions; and a processor configured to call the instructions from the memory and enable the implementation of the image processing method based on artificial intelligence according to the above when executing the instructions.

[0018] The third aspect of the present application provides an image processing system based on artificial intelligence, comprising: an image acquisition device for acquiring a calibration image of a calibration device; the calibration device comprises a pattern board with multiple patterns, and the color of the patterns on the pattern board in the calibration image is different from the color of other areas; and the image processing device based on artificial intelligence according to the above.

[0019] In the embodiments of the present application, the calibration device further comprises a heating plate connected to the pattern board, so that each pattern in the calibration image presents a clear contrast with the background when the heating plate is heated; the image acquisition device comprises an infrared camera and a visible light camera; and the multiple patterns are multiple hollow patterns which are the same and arranged in an array.

[0020] The fourth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, the instructions being used to cause a machine to execute the image processing method based on artificial intelligence according to the above.

[0021] Through the above technical solutions, the embodiments of the present application design the pattern board of the calibration device to have multiple patterns, acquire a calibration image of the calibration device, perform feature extraction on the calibration image to obtain a first feature map, and convert the image data in the calibration image into a form which is easy to process and analyze. A deep learning architecture combining a bounding box regression network and a center regression network is constructed, multiple regions of interest in the first feature map are extracted through the deep learning architecture, a pre-designed preset prior box is combined, and each candidate region of interest is output to obtain the coordinates of the image center point of each pattern in the calibration image. The above method can obtain clear imaging edge information, and in the case of edge blur and poor imaging quality in the calibration image, more accurate candidate regions are obtained to significantly improve the detection accuracy of image recognition.

[0022] Other features and advantages of the embodiments of the present application will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used to explain the embodiments of the present application together with the following specific implementation, but do not constitute a limitation to the embodiments of the present application. In the drawings:

[0024] FIG. 1 schematically shows a flowchart of an image processing method based on artificial intelligence according to an embodiment of the present application;

[0025] FIG. 2 schematically shows a flowchart of generating a first feature map according to an embodiment of the present application;

[0026] FIG. 3 schematically shows a network structure of a center regression network and a bounding box regression network according to an embodiment of the present application;

[0027] FIG. 4 schematically shows a flowchart of an artificial intelligence-based image processing method according to an embodiment of the present application;

[0028] FIG. 5 schematically shows a flowchart of a PRPN network algorithm according to an embodiment of the present application;

[0029] FIG. 6 schematically shows a structural block diagram of an artificial intelligence-based image processing system according to an embodiment of the present application;

[0030] FIG. 7 schematically shows a structural diagram of a calibration device according to an embodiment of the present application;

[0031] FIG. 8 schematically shows a flowchart of determining camera intrinsic parameters according to an embodiment of the present application;

[0032] FIG. 9 schematically shows a structural block diagram of an artificial intelligence-based image processing device according to an embodiment of the present application;

[0033] FIG. 10 schematically shows a structural diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the specific embodiments described herein are merely used to explain and illustrate the embodiments of the present application and should not be used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0035] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are merely used to explain the relative positional relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly. In addition, it should be noted that the “power”, “electricity” and “energy” involved in the embodiments of the present application can be understood interchangeably.

[0036] In addition, if the description of "first", "second", etc. is involved in the embodiments of the present application, the description of "first", "second", etc. is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the protection scope claimed by the present application.

[0037] Fig. 1 schematically shows a flow diagram of an image processing method according to an embodiment of the present application. As shown in Fig. 1, the present application provides an image processing method based on artificial intelligence, which can include the following steps.

[0038] S102, collecting a calibration image of a calibration device, the calibration device comprising a plurality of patterns of a pattern plate.

[0039] It can be understood that the calibration device refers to a device for determining the measurement error of a sensor or a measurement system and correcting it. In the embodiments of the present application, the calibration device is a device for calibrating the image acquisition device. The calibration device comprises a pattern plate, and the pattern plate has a plurality of patterns. The patterns on the pattern plate are different from other areas in shape, or color, reflectivity, temperature. Specifically, the image acquisition device can be an optical camera, an infrared camera, a scanner, etc. Understandably, the calibration image refers to the image collected by the image acquisition device for the calibration device. If the image acquisition device is an optical camera, the calibration image is an optical image, and if the image acquisition device is an infrared camera, the calibration image is an infrared image. Wherein, the calibration of optical image is based on the difference between the pattern and other areas in shape and color, and the calibration of infrared image is based on the difference between the pattern and other areas in temperature. The color of the pattern on the pattern plate is different from the shape, color, reflectivity, temperature of other areas, which can distinguish the pattern on the pattern plate and other areas on the image in the calibration image, and obtain clear imaging edge information, so as to process the collected calibration image, effectively complete the calibration of the image acquisition device, and improve the precision and efficiency of the device calibration.

[0040] S104, performing feature extraction on the calibration image to obtain a first feature map after processing.

[0041] Image feature extraction refers to extracting representative features from original image data for subsequent image analysis and processing tasks. In the embodiments of the present application, the calibration image contains information of the pattern on the pattern plate. By extracting features in the calibration image, a processed first feature map is obtained. The first feature map includes key attribute information such as color and shape in the calibration image.

[0042] In the embodiments of the present application, processing the calibration image to obtain the processed first feature map includes: inputting the calibration image into a residual network and outputting multiple second feature maps of different scales through the residual network; inputting the multiple second feature maps into an upsampling module and outputting multiple third feature maps of the same size through the upsampling module; connecting all the third feature maps to obtain a fourth feature map; inputting the fourth feature map into an attention mechanism module and outputting the first feature map after spatial dimension and channel dimension pooling processing of the fourth feature map through the attention mechanism module.

[0043] It can be understood that the residual network is a convolutional neural network, and each layer in the residual network learns the residual between the input and the output, i.e., the goal of network learning is to map the input to the difference between the expected output and the input, optimize the network parameters, and reduce the training difficulty. The calibration image is input into the residual network, and multiple second feature maps are output through the residual network. The residual network can output corresponding second feature maps at different stages for the calibration image. Different second feature maps contain feature information of different scales of the calibration image. Further, multiple second feature maps are input into an upsampling module, and each second feature map is processed by upsampling. It can be understood that in image processing, upsampling processing refers to a technique for increasing the resolution of an image or recovering missing information.

[0044] Specifically, the upsampling processing methods include but are not limited to nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos interpolation, and magnification factor method. In the embodiments of the present application, after each second feature map is processed by upsampling, each second feature map is processed to the same size, so that multiple third feature maps of the same size are output through the upsampling module. Further, all third feature maps can be connected to connect the features in the map to obtain a fourth feature map. Specifically, all third feature maps can be connected by feature connection to obtain the fourth feature map. The fourth feature map includes feature information of all third feature maps.

[0045] Further, the fourth feature map is input to an attention mechanism module. The attention mechanism module includes a channel attention mechanism (CAM) and a spatial attention mechanism (SAM). The channel attention mechanism can better understand different features in the image, and by calculating the importance of each channel, the model can distinguish the features between different channels. The spatial attention mechanism focuses on specific regions in the image, and by weighting the importance of each pixel in space, the model can better capture the spatial structure in the image. The fourth feature map can be first processed by the channel attention mechanism for channel dimension pooling, and then further processed by the spatial attention mechanism for spatial dimension pooling, to obtain the processed first feature map, which can effectively fuse the features of different scales and channels of the image. The above scheme for extracting the first feature map can effectively eliminate image noise caused by external environment and other factors.

[0046] FIG. 2 schematically shows a flowchart of generating a first feature map according to an embodiment of the present application. As shown in FIG. 2, a residual network ResNet50 is used, and second feature maps are respectively output by stage1, stage2, stage3 and stage4 of the ResNet50. After each second feature map is processed by an unpooling module, four third feature maps with the same size are obtained. They are F1, F2, F3 and F4 respectively. The four feature maps with the same size are spliced together in the channel dimension by feature connection to obtain a fourth feature map, thereby fusing the four feature maps of different scales to make full use of information of different scales. Further, the fourth feature map is sequentially processed by a channel attention mechanism CAM and a spatial attention mechanism SAM to obtain a first feature map.

[0047] Specifically, the channel attention mechanism can better understand different features in the image, and by calculating the importance of each channel, the model can distinguish the features between different channels. The thinking process of the channel attention is to perform global maximum pooling and global average pooling on an input feature map with a size of HxWxC in the spatial dimension, and output two 1x1xC feature maps. Then, the results of the global maximum pooling and the global average pooling are respectively input into a shared multi-layer perceptron (MLP) to learn two 1x1xC feature maps. The number of neurons in the first layer of the MLP is C / r, r is a dimension reduction coefficient, the activation function is Relu, and the number of neurons in the second layer is C. The MLP learns the features in the channel dimension and the importance of each channel. Finally, the results output by the MLP are subjected to an Add operation, and after mapping processing by a Sigmoid activation function, a channel attention weight matrix M c is finally obtained.

[0048] M c(F a ) = σ(MLP(AvgPool(F a ) + MLP(MaxPool(F a ))) (2)

[0049] wherein F a is a feature map input to the channel attention mechanism, M c (F a ) is a channel attention weight matrix, σ is a sigmoid activation function, MLP is a multi-layer perceptron, AvgPool is an average pooling layer, and MaxPool is a maximum pooling layer.

[0050] The spatial attention module focuses on specific regions in the image, and weights the importance of each pixel in the space, so that the model can better capture the spatial structure in the image. For an input feature map with a size of HxWxC, global maximum pooling and global average pooling are performed in the channel dimension to obtain two feature maps with a size of HxWx1. The results of global maximum pooling and global average pooling are concatenated to obtain a feature map with a size of HxWx2. An m x m convolution operation is performed on the concatenated result to obtain a feature map with a size of HxWx1, and a spatial attention weight matrix M s is obtained through a sigmoid activation function. Specifically, the function expression of the spatial attention mechanism is shown in the following formula (3):

[0051] M s (F b ) = σ(f m×m ([AvgPool(F b ); MaxPool(F b )])) (3)

[0052] wherein F b is a feature map input to the spatial attention mechanism, M s (F b ) is a spatial attention weight matrix, f m×m is an m x m convolution layer, σ is a sigmoid activation function, AvgPool is an average pooling layer, and MaxPool is a maximum pooling layer.

[0053] S106, input the first feature map to the bounding box regression network to output a plurality of regions of interest corresponding to the calibration device in the first feature map through the bounding box regression network.

[0054] It can be understood that the bounding box regression network enables the model to more accurately predict the bounding box of the object by learning and predicting correction values for the initial object position (usually an initial bounding box). These correction values include horizontal offset, vertical offset, width scaling, and height scaling, all relative to the initial bounding box. Inputting the first feature map into the bounding box regression network can fine-tune the initial bounding box to make it closer to the true position of the calibration device in the first feature map. That is, the bounding box regression network outputs multiple regions of interest corresponding to the calibration device in the first feature map. The region of interest refers to the region in the first feature map where the calibration device is most likely to exist. In the embodiments of the present application, identifying the feature map that has been removed from image noise can effectively improve the identification accuracy of the region of interest.

[0055] In S108, a preset number of candidate boxes are generated for each region of interest based on the preset prior box, and the size of each candidate box is the same as that of the preset prior box. The preset number is determined according to the number of the plurality of patterns.

[0056] The preset prior box refers to a bounding box that is preset for an image to be identified in a target detection technology. The size of the preset prior box is preset. Specifically, the preset prior box can be set according to the size of the pattern on the calibration device. Understandably, the region of interest is used to lock the region in the first feature map where the calibration device may exist, and the candidate box is used to lock the region in the region of interest where the pattern on the pattern plate may exist. Based on the preset prior box, a plurality of candidate boxes can be generated for each region of interest. The preset number refers to the preset number of generated candidate boxes for each region of interest, which can be determined according to the number of patterns on the pattern plate. For example, the preset number can be the number of patterns on the pattern plate. By using the preset prior box, a large number of candidate regions can be sampled, and it can be determined whether these regions contain the target, so that the real bounding box of the target can be more quickly and accurately predicted.

[0057] In the embodiments of the present application, the plurality of patterns are a plurality of identical and regularly arranged patterns. Generating a preset number of candidate boxes for each region of interest based on the preset prior box includes: obtaining the size of the circumscribed rectangle of any one pattern; generating a preset prior box according to the size of the circumscribed rectangle; inputting the preset prior box and the region of interest into a region generation module for each region of interest to output a preset number of candidate boxes for each region of interest through the region generation module.

[0058] It can be understood that, since the plurality of patterns on the pattern plate are all the same and arranged regularly, the skilled person can directly obtain the size of the circumscribed rectangle of any one pattern when designing the calibration device, i.e. the width and height of the circumscribed rectangle. The size of the circumscribed rectangle is scaled to generate a preset prior box matching the first feature map. For each region of interest, the preset prior box and the region of interest are input to the region generation module to output a preset number of candidate boxes for each region of interest through the region generation module. The region generation module refers to a module for generating a candidate box for a region of interest through a preset prior box. By designing a preset prior box through the size of the circumscribed rectangle of the pattern, a candidate box facilitating the identification of the target can be quickly generated.

[0059] In an embodiment of the present application, the function expression of the region generation module is shown in the following formula (1):

[0060] wherein, Proposal (k,n) refers to the nth candidate box of the kth region of interest, bbox k refers to the kth region of interest, Base_anchor n refers to the nth candidate box generated based on the preset prior box, x1, x2 respectively refer to the horizontal coordinates of the diagonal points of the circumscribed rectangle corresponding to the preset prior box, y1, y2 respectively refer to the vertical coordinates of the diagonal points of the circumscribed rectangle corresponding to the preset prior box, x, y refer to the horizontal and vertical coordinates of the fixed point of each region of interest, w, h respectively refer to the width and height of each region of interest.

[0061] S110, input all candidate boxes and the first feature map to the center regression network to obtain the coordinates of the image center point of each pattern in the calibration image.

[0062] It can be understood that the center regression network is similar to the bounding box regression network, but the center regression network is for identifying the center point of the pattern on the pattern plate in the candidate box. The image center point of the pattern refers to the center point corresponding to the imaging of the pattern on the pattern plate in the calibration image. After inputting all candidate boxes and the first feature map to the center regression network, the center regression network can identify each candidate box in the first feature map, thereby predicting the probability of the existence of the image center point of the pattern in each candidate box and the coordinates of the image center point of the pattern, to further determine the coordinates of the image center point of each pattern in the calibration image. Specifically, the network structure of the center regression network and the bounding box regression network is shown in FIG. 3. From the input layer to the output layer, both the center regression network and the bounding box regression network pass through 5 convolution operations of 3x3, and use the Relu function as the activation function.

[0063] By the technical solution, the pattern plate of the calibration device is designed to have multiple patterns, a calibration image is collected for the calibration device, and then feature extraction is performed on the calibration image to obtain a first feature map, so that the image data in the calibration image is converted into a form that is easy to process and analyze. A deep learning architecture combining a bounding box regression network and a center regression network is constructed, multiple regions of interest in the first feature map are extracted through the deep learning architecture, a pre-designed preset prior box is combined, and each candidate region of interest is output to obtain the coordinates of the image center point of each pattern in the calibration image. The above method can obtain clear imaging edge information, and in the case of edge blur and poor imaging quality in the calibration image, more accurate candidate regions are obtained to significantly improve the detection accuracy of the image center point of the pattern.

[0064] In the embodiments of the present application, inputting all candidate boxes into the center regression network to determine the center point of each pattern in the calibration image according to the predicted image coordinates output by the center regression network comprises: inputting the first feature map and all candidate boxes into the center regression network to output multiple predicted image coordinates and a predicted probability corresponding to each preset image coordinate according to the center regression network; arranging the multiple predicted probabilities from high to low, and determining the predicted image coordinates corresponding to the preset number of predicted probabilities arranged in the front as the first image coordinates of the image center point of each pattern in the calibration image.

[0065] It can be understood that the predicted image coordinates refer to the predicted coordinates of the image center point of the pattern on the pattern plate for each candidate box in the first feature map, and the coordinate system is the image coordinate system in which the calibration image is located. The predicted probability refers to the probability of the image center point of the pattern on the pattern plate for each candidate box in the first feature map. Each candidate box is output with a predicted image coordinate and a predicted probability by the center regression network. The number of patterns on the pattern plate of the calibration device is fixed, and according to the foregoing scheme, the number of candidate boxes in each region of interest is determined according to the number of patterns on the pattern plate. Therefore, the number of predicted image coordinates determined according to the candidate boxes is greater than or equal to the number of patterns on the pattern plate. Then, the obtained predicted probabilities can be arranged from high to low, a preset number of predicted probabilities arranged in the front can be selected, and each predicted image coordinate corresponding to these predicted probabilities can be determined as the first image coordinates of the image center point of each pattern in the calibration image.

[0066] In a specific embodiment, as shown in FIG. 4, a residual network ResNet50 is adopted, and second feature maps are respectively output by stage1, stage2, stage3 and stage4 of ResNet50. After each second feature map is processed by an upsampling module, four third feature maps with the same size are obtained. They are F1, F2, F3 and F4 respectively. The four feature maps with the same size are spliced together in the channel dimension by feature connection to obtain a fourth feature map F5, so as to fuse the four feature maps of different scales to make full use of information of different scales. Further, the fourth feature map is sequentially processed by a channel attention mechanism CAM and a spatial attention mechanism SAM to obtain a first feature map F6, which is input into a PRPN (Preset Region Proposal Network) network. The PRPN network is also used to generate a candidate region. Specifically, as shown in FIG. 5, in the PRPN network, the first feature map F6 is input into a BoxRegression network, and after maximum suppression processing by NMS, k regions of interest of the first feature map F6 are output. The output channel of the BoxRegression network is k x 4, including (x, y, w, h) of k bboxes, x, y, w and h being the left upper point horizontal coordinate, vertical coordinate, width and height of each region of interest. Further, according to the size of the bounding rectangle of any pattern, a preset prior box is generated. Based on the preset prior box, the preset prior box and the region of interest are input into a RegionGenerate module, so as to generate n candidate boxes for each region of interest by the RegionGenerate module, n being the total number of patterns on the pattern plate. Further, after the n candidate boxes generated above are input into a ROIPooling layer for pooling processing, they are input into a CenterRegression network to output the predicted image coordinates and the predicted probability of each candidate box. The predicted image coordinates of the top n predicted probabilities are taken as the first image coordinates of the image center points of the patterns in the calibration image. The output channel of the CenterRegression network is n x k x 3, including (x, y, scores) of n points, x, y and scores being the predicted image horizontal coordinate, vertical coordinate and predicted probability. By the above method, more accurate candidate regions can be obtained in the case of a calibration image with blurred edges and poor imaging quality, and the detection accuracy of identifying the pattern center point can be significantly improved.

[0067] In the embodiments of the present application, the method further comprises: sorting all the actual center points of the patterns on the pattern board, and obtaining the relative positions between all the sorted actual center points; determining the world coordinates of each sorted actual center point according to the relative position corresponding to each sorted actual center point; performing perspective transformation on the first image coordinates of all the image center points based on the relative positions of all the sorted actual center points, to determine the first image coordinates corresponding to each sorted actual center point; and determining the intrinsic parameters of the image acquisition device for collecting the calibration image according to all the first image coordinates and world coordinates based on the corresponding relationship between the first image coordinates and the world coordinates of each sorted actual center point.

[0068] Camera intrinsic calibration is an important step in computer vision, aiming to determine the internal parameters of the camera, including focal length, optical center and distortion parameters, etc. These parameters can play an important role in subsequent image processing tasks, such as stereo vision, three-dimensional reconstruction and object tracking, etc. The actual center point refers to the center point of the pattern on the pattern board of the calibration device in the real space. Understandably, the image center point and the actual center point are relative, and the image center point is the projection of the actual center point in the calibration image. Through the foregoing scheme, after obtaining the first image coordinates of the image center points of each pattern in the calibration image, the intrinsic parameters of the image acquisition device can be further calibrated through the corresponding relationship between the first image coordinates of the image center points and the world coordinates of the actual center points. However, the first image coordinates of these image center points obtained by the foregoing scheme are an unordered data set, and the corresponding relationship between the first image coordinates of each image center point and the world coordinates of each actual center point cannot be determined. Therefore, it is necessary to first determine the sorting of the first image coordinates of the image center points, and each sorted image center point obtains a corresponding number, which enables each image center point to be distinguished from other image center points, so that the world coordinates of the corresponding actual center point can be found through the number.

[0069] Specifically, the actual center points of all the patterns on the pattern plate can be sorted first. Since the positions of the patterns on the pattern plate are fixed, the relative positions between the actual center points of all the patterns on the pattern plate can be obtained. Further, after the relative positions between the actual center points are known, the world coordinates of each actual center point can be directly derived, and the coordinate system of the world coordinates can take any fixed position in the space where the calibration device is located as the origin. Moreover, after the relative positions between the actual center points and the order of the actual center points are known, the first image coordinates of all the image center points can be perspective transformed based on the relative positions of all the sorted actual center points, the correspondence between the actual center points and the image center points is found, and the first image coordinates corresponding to each sorted actual center point are determined. Further, based on the corresponding relationship between the first image coordinates and the world coordinates of each sorted actual center point, the intrinsic parameters of the image acquisition device for collecting the calibration image are determined according to all the first image coordinates and the world coordinates. For example, the intrinsic parameters of the image acquisition device can be calculated by using Zhang Zhengyou calibration method.

[0070] In the embodiments of the present application, the arrangement mode of the plurality of patterns is an array, the relative positions include arrangement intervals of the array, the number of rows of the array is a first quantity, the number of columns of the array is a second quantity, and sorting the actual center points of all the patterns includes: sorting the actual center points of all the patterns according to the order of the array from row to column; and perspective transforming the first image coordinates of all the image center points based on the relative positions of all the sorted actual center points to determine the first image coordinates corresponding to each sorted actual center point includes: after determining the first image coordinates of each image center point, determining two opposite corners in all the image center points according to all the first image coordinates; determining a plurality of first corners for constructing a first target polygon in the calibration image according to the two opposite corners; determining a plurality of second corners corresponding to the shape of the pattern plate according to the arrangement intervals, the first quantity and the second quantity of the array, so as to construct a second target polygon corresponding to the shape of the pattern plate in the calibration image through the plurality of second corners; generating a second image coordinate corresponding to each sorted actual center point in the second target polygon based on the arrangement intervals of the array; determining a perspective transformation matrix between the first target polygon and the second target rectangle according to the plurality of first corners and the plurality of second corners; converting each first image coordinate through the perspective transformation matrix to determine the second image coordinate corresponding to each first image coordinate, so as to determine the first image coordinates corresponding to each sorted actual center point.

[0071] In the embodiments of the present application, determining the first plurality of corner points of the first target polygon in the calibration image according to the two diagonal points comprises: for any one of the two diagonal points, determining a line between each image center point and the diagonal point to determine the number of patterns passing through each line; for each diagonal point, the line passing through the pattern is the first target line if the number of patterns passing through the line is a first number, and the line passing through the pattern is the second target line if the number of patterns passing through the line is a second number; for each diagonal point, determining the intersection point of the first target line and the second target line; and determining all the diagonal points and all the intersection points of the lines as the first plurality of corner points of the first target polygon in the calibration image.

[0072] Specifically, in order to perform perspective transformation on the first image coordinates of all image center points, on the one hand, a view plane corresponding to all image center points, i.e., the first target polygon, needs to be constructed, and on the other hand, a view plane corresponding to all actual center points, i.e., the second target polygon, needs to be constructed. In order to facilitate the calibration of the pattern center points, the skilled person can design the patterns on the pattern plate of the calibration device in the form of an array, and obtain the number of rows and the number of columns of the array, wherein the number of rows of the array is a first number, and the number of columns of the array is a second number. After determining the first image coordinates of each image center point, the image center points are grouped two by two, the distance between the two image center points in the calibration image is calculated according to the first image coordinates of the two image center points, and the group of image center points with the maximum distance is selected as the diagonal points. Taking any one of the diagonal points, the lines between the center points of each pattern and the diagonal point are determined in turn, and the number of patterns passing through the line, i.e., the number of connected domains formed by the line passing through the patterns, is determined. The line passing through the pattern is the first target line if the number of patterns passing through the line is the first number. In order to facilitate understanding, it can be assumed that the array is a rectangle, if the number of patterns passing through the line between a certain image center point and the diagonal point is the number of rows of the array, then this image center point is most likely to be a corner point of the rectangle, and the first target line is most likely to be a side of the rectangle. Similarly, the second target line passing through the pattern by a second number is also most likely to be another side of the rectangle. Further, the intersection points of the lines between all the first target lines and all the second target lines are determined, i.e., the intersection points of the first target line of one diagonal point and the first target line and the second target line of another diagonal point, and there can be multiple intersection points between multiple lines. These intersection points are the corner points of the array, and of course also include the diagonal points mentioned above. The first corner point refers to the corner point of the first target polygon. All the diagonal points and the intersection points of the lines are determined as the first corner points, and these first corner points are sorted in a clockwise order and connected in turn to construct the first target polygon.

[0073] Further, the maximum distance between the array arrangement intervals of the pattern plate is found, and based on the maximum distance, the number of rows and the number of columns of the array, a plurality of second corner points corresponding to the shape of the pattern plate are determined, so as to construct a second target polygon corresponding to the shape of the pattern plate in the calibration image by the plurality of second corner points. That is, one view plane corresponding to all the actual center points is constructed in the calibration image. Moreover, in the second target polygon, the second image coordinates corresponding to each of the ordered actual center points can be generated in the calibration image, with the arrangement interval of the array and the order of the actual center points being known. Then, based on the plurality of first corner points and the plurality of second corner points, the perspective transformation matrix between the first target polygon and the second target rectangle, i.e., the perspective transformation matrix between the two view planes, can be determined. After each first image coordinate is converted by the perspective transformation matrix, the second image coordinate corresponding to the perspective-transformed coordinate of the first image coordinate can be found. That is, the second image coordinate corresponding to each first image coordinate is found. Therefore, after the order of the actual center points corresponding to the second image coordinates is known, the first image coordinates corresponding to each of the ordered actual center points can be determined. Then, the coordinates of the pattern center points of the two view planes can be one-to-one corresponding, and the correspondence between the first image coordinates and the world coordinates can be determined. Then, based on the above correspondence, the intrinsic parameters of the image acquisition device for acquiring the calibration image can be determined based on all the first image coordinates and the world coordinates.

[0074] FIG. 6 schematically shows a structural block diagram of an artificial intelligence-based image processing system according to an embodiment of the present application. As shown in FIG. 6, an artificial intelligence-based image processing system is provided, which includes:

[0075] an image acquisition device 610 configured to acquire a calibration image of a calibration device;

[0076] the calibration device 620 includes a pattern plate having a plurality of patterns;

[0077] the image processing device 630 is configured to execute the artificial intelligence-based image processing method described above.

[0078] Specifically, the image acquisition device can be an optical camera, an infrared camera, a scanner, etc. The calibration image can be an optical image or an infrared image. The calibration device includes a pattern board having a plurality of patterns. The calibration device can be captured by the image acquisition device, and the patterns on the pattern board are distinguished from other areas in shape, color, reflectivity, or temperature. For example, when the image acquisition device is an infrared camera, the patterns in the captured infrared image are distinguished from the background in brightness. When the image acquisition device is an optical camera, the patterns in the captured optical image are distinguished from the background in color contrast. The image processing device can process the captured calibration image by the image processing method to obtain the coordinates of the image center of the patterns.

[0079] In an embodiment of the present application, the calibration device further includes a heating plate connected to the pattern board, so that when the heating plate is heated, each pattern in the calibration image is distinguished from the background in a clear contrast. The image acquisition device includes an infrared camera and a visible light camera, and the plurality of patterns are a plurality of hollow patterns arranged in an array.

[0080] As shown in FIG. 7, the calibration device includes a heating plate 10 and a pattern board 20, and the pattern board 20 is designed with a plurality of hollow patterns 30. The heating plate has high structural strength and is not easy to bend or deform. The heating plate also has a temperature control system that can stabilize the temperature at a set value. The heating plate is rectangular in shape, and four mounting holes are left at the corners for connecting the heating plate and the pattern board. The size and mounting holes of the pattern board are consistent with those of the heating plate, and the hollow patterns are regularly arranged circles with a number of at least 4. The heating plate and the pattern board are fixed by bolts, and a gap is left between the two plates instead of being tightly attached together. After the two plates are fixed, the power supply is connected to use. After the power is turned on, the heating plate generates heat, and in the calibration image captured by the infrared thermal imaging camera, the hollow patterns of the pattern board are highlighted. In addition, the heating plate and the pattern board are respectively two colors with clear contrast, and in the calibration image captured by the visible light camera, the hollow patterns are distinguished from the background in a clear contrast.

[0081] Figure 8 schematically shows a flow diagram for determining camera intrinsic parameters according to an embodiment of the present application. As shown in Figure 8, the method for determining camera intrinsic parameters is applied to the image acquisition system shown in Figure 6, and uses the calibration device shown in Figure 7. First, the calibration device and the camera are powered on, with a temperature setting of T1 °C, which is higher than the ambient temperature, and the calibration device is placed on the ground. The camera is installed vertically downward, and after the camera is powered on, a calibration image is acquired. The installation position of the camera is adjusted so that the calibration device occupies about one third of the field of view of the camera. The calibration device is moved, and at the same time, calibration images are acquired. The position of the calibration device is adjusted so that the calibration device appears in each region of the field of view of the camera, and the number of images acquired is about 20 to form an image set. For each calibration image, the first image coordinates of the image center points of the circles in the calibration image are calculated by the deep learning algorithm in the above image processing method. Further, the first image coordinates are sorted by the aforementioned method, i.e., the order of the center points of the circles is obtained, so as to correspond the first image coordinates to the world coordinates of the actual center points, so as to calculate the intrinsic parameters of the camera by Zhang Zhengyou's calibration method.

[0082] Step 1, calculate the distance between each pair of image center points, and take the maximum distance distance max as a pair of diagonal points p1, p2.

[0083] Step 2, take one of the diagonal points p_diagonal, and sequentially find the straight line L i composed of p_diagonal and other image center points, and the number N i of connected domains of the hollow pattern crossed by L li .

[0084] Step 3, given that the number of regularly arranged circles is r rows and c columns, take the straight line L li with N r = r, and take the straight line L li with N c = c.

[0085] Step 4, perform steps 2) and 3) on the two diagonal points to obtain two sets of straight lines L r1 , L c1 and L r2 , L c2 , find the intersection point p3 of L r1 and L c2 , and the intersection point p4 of L r2 and L c1 .

[0086] Step 5, reorder p1, p2, p3, and p4 in clockwise order to obtain the four corner points corner src of the quadrilateral polygon src in which the image center points are located.

[0087] Step 6, constructing a rectangle polygon real , wherein the four corner point coordinates corner real are (0, 0), Based on the arrangement of the hollow pattern, the second image coordinates center real of each sorted actual center point in the polygon real are calculated in the order of row first and column second.

[0088] Step 7, the perspective transformation matrix M p from the quadrilateral polygon src to the rectangular polygon real is calculated according to corner src and corner real .

[0089] Step 8, the first image coordinates of the image center points are converted by the perspective transformation matrix M p to obtain new coordinates, which correspond to the second image coordinates center real of each sorted actual center point, so as to determine the corresponding first image coordinates of each sorted actual center point.

[0090] Step 9, the world coordinates center world of each sorted actual center point are generated in the order of row first and column second according to the arrangement interval of the hollow pattern.

[0091] Step 10, the intrinsic parameters of the camera are calculated by Zhang Zhengyou calibration method according to the world coordinates center world and the first image coordinates of each sorted actual center point.

[0092] FIG. 9 schematically shows a structural block diagram of an image processing apparatus based on artificial intelligence according to an embodiment of the present application. As shown in FIG. 9, the image processing apparatus based on artificial intelligence according to an embodiment of the present application can comprise:

[0093] a memory 910 configured to store instructions; and

[0094] a processor 920 configured to call the instructions from the memory 910 and capable of realizing the image processing method described above when executing the instructions.

[0095] Specifically, in the embodiment of the present application, the processor 920 can be configured to:

[0096] collect a calibration image of a calibration device, the calibration device comprising a pattern board having a plurality of patterns;

[0097] perform feature extraction on the calibration image to obtain a processed first feature map;

[0098] input the first feature map into the bounding box regression network to output, by the bounding box regression network, a plurality of regions of interest corresponding to the calibration device in the first feature map;

[0099] generate, based on a preset anchor box, a preset number of candidate boxes for each region of interest, each candidate box having a same size as the preset anchor box, and the preset number being determined according to a number of the plurality of patterns;

[0100] input all the candidate boxes and the first feature map into the center regression network to obtain coordinates of image center points of each pattern in the calibration image.

[0101] In the embodiments of the present application, the processor 920 can also be configured to:

[0102] input the first feature map and all the candidate boxes into the center regression network to output, by the center regression network, a plurality of predicted image coordinates and a predicted probability corresponding to each preset image coordinate; arrange the plurality of predicted probabilities from high to low, and determine, as first image coordinates of the image center points of each pattern in the calibration image, predicted image coordinates corresponding to a preset number of predicted probabilities arranged in front.

[0103] In the embodiments of the present application, the plurality of patterns are a plurality of same and regularly arranged patterns, and the processor 820 can also be configured to:

[0104] obtain a size of a circumscribed rectangle of any one pattern; generate a preset anchor box according to the size of the circumscribed rectangle; and input, for each region of interest, the preset anchor box and the region of interest into the region generation module to output, by the region generation module, a preset number of candidate boxes for each region of interest.

[0105] In the embodiments of the present application, the processor 920 can also be configured to:

[0106] The function expression of the region generation module is shown in the following formula (1):

[0107] wherein, Proposal (k,n) refers to the nth candidate box of the kth region of interest, bbox k refers to the kth region of interest, Base_anchor nx1, x2, y1, y2 respectively refer to horizontal coordinates of opposite corners of a preset prior box corresponding to an external rectangle, y1, y2 respectively refer to vertical coordinates of opposite corners of a preset prior box corresponding to an external rectangle, x, y respectively refer to horizontal coordinates and vertical coordinates of fixed points of each region of interest, w, h respectively refer to width and height of each region of interest.

[0108] In the embodiment of the present application, the processor 920 can also be configured to:

[0109] input the calibration image into the residual network, and output a plurality of second feature maps with different scales through the residual network; input the plurality of second feature maps into the up-sampling module, and output a plurality of third feature maps with the same size through the up-sampling module; connect all the third feature maps to obtain a fourth feature map; input the fourth feature map into the attention mechanism module, and output a first feature map after spatial dimension and channel dimension pooling processing of the fourth feature map through the attention mechanism module.

[0110] In the embodiment of the present application, the processor 920 can also be configured to:

[0111] sort all the actual center points of the patterns on the pattern plate, and obtain the relative positions between all the sorted actual center points; determine the world coordinates of each sorted actual center point according to the relative position corresponding to each sorted actual center point; perform perspective transformation on the first image coordinates of all the image center points based on the relative positions of all the sorted actual center points, to determine the first image coordinates corresponding to each sorted actual center point; determine the intrinsic parameters of the image acquisition device for collecting the calibration image according to all the first image coordinates and world coordinates based on the corresponding relationship between the first image coordinates and the world coordinates of each sorted actual center point.

[0112] In the embodiment of the present application, the arrangement mode of the plurality of patterns is an array, the relative positions include arrangement intervals of the array, the number of rows of the array is a first quantity, and the number of columns of the array is a second quantity, and the processor 920 can also be configured to:

[0113] sort the actual center points of all the patterns according to the order of first row and then column based on the array; and

[0114] After determining the first image coordinates of each image center point, two diagonal points among all the image center points are determined according to all the first image coordinates; a plurality of first corner points of a first target polygon constructed in the calibration image are determined according to the two diagonal points; a plurality of second corner points corresponding to the shape of the pattern plate are determined according to the arrangement interval, the first number and the second number of the array, so as to construct a second target polygon corresponding to the shape of the pattern plate in the calibration image through the plurality of second corner points; the second image coordinates corresponding to each ordered actual center point are generated in the second target polygon based on the arrangement interval of the array; the perspective transformation matrix between the first target polygon and the second target rectangle is determined according to the plurality of first corner points and the plurality of second corner points; each first image coordinate is converted through the perspective transformation matrix to determine the second image coordinates corresponding to each first image coordinate, so as to determine the first image coordinates corresponding to each ordered actual center point.

[0115] In the embodiments of the present application, the processor 920 can also be configured to:

[0116] For any one of the two diagonal points, a line between each center point and the diagonal point is determined to determine the number of patterns passing through each line; for each diagonal point, the line passing through the pattern for the first number is the first target line, and the line passing through the pattern for the second number is the second target line; the intersection points of all the first target lines and all the second target lines are determined; all the diagonal points and all the intersection points of the lines are determined as a plurality of first corner points for constructing the first target polygon in the calibration image.

[0117] According to the technical solution, the pattern plate of the calibration device is designed to have multiple hollow patterns, and a heating plate with a different color is superimposed on the pattern plate. After the heating plate is powered on, the color of the pattern on the pattern plate is different from the color and temperature of other areas in the calibration image collected by the image acquisition device, so that the pattern and the background can be clearly distinguished. Then, a residual network is used to extract features from the calibration image to obtain a first feature map, and the image data in the calibration image is converted into a form that is easy to process and analyze. A deep learning architecture combining a bounding box regression network and a center regression network is constructed, and multiple regions of interest in the first feature map are extracted through the deep learning architecture. In combination with a preset prior box designed based on the size of the pattern, each candidate region of interest is output, and the predicted image coordinates and the prediction probability of each candidate region are output. According to the prediction probability and the number of patterns, the coordinates of the image center point of each pattern in the calibration image are obtained. The above method can obtain more accurate candidate regions in the case of edge blur and poor imaging quality of the calibration image, thereby significantly improving the detection accuracy of the image center point of the pattern. Further, a view plane is constructed through the actual relative position of the pattern plate, and another view plane is constructed through the image center point coordinates of the calibration image, and a perspective conversion matrix between the two view planes is obtained. In this way, the first image coordinates of each center point can be corresponded to the world coordinates, so that the intrinsic parameters of the camera can be obtained through the Zhang Zhengyou calibration method. The above method enables imaging of the calibration image from the design structure of the calibration device, and uses a unified algorithm to calibrate visible light cameras and infrared thermal imaging cameras. The combination of the two can not only obtain clear edge information, but also obtain more accurate candidate regions in the case of edge blur and poor imaging quality of the calibration image, thereby significantly improving the detection accuracy of image recognition.

[0118] The embodiment of the present application also provides a machine readable storage medium, which stores instructions for causing a machine to execute the above-mentioned image processing method based on artificial intelligence.

[0119] In an embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 10. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operating system B01 and the computer program B02 in the non-volatile storage medium A04 to run. The database of the computer device is configured to store artificial intelligence-based image processing data. The network interface A02 of the computer device is configured to communicate with an external terminal through a network connection. The computer program B02 is executed by the processor A01 to implement an artificial intelligence-based image processing method.

[0120] Those skilled in the art can understand that the structure shown in FIG. 10 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] Those skilled in the art can understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0123] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flowsheets and / or block or blocks of the block diagrams.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheets and / or block or blocks of the block diagrams.

[0125] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0126] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, or other memory technologies, about which the processor can execute instructions. The memory is an example of computer readable media.

[0127] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0128] It should also be noted that the terms "comprising", "comprises" or other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0129] The above embodiments are only used to illustrate the present application, but not to limit it. Instead of the above, various modifications and changes can be made to the application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall into the scope of the claims of the application.

Claims

1. An image processing method, the method comprising: capturing a calibration image of a calibration device, the calibration device comprising a pattern board having a plurality of patterns; performing feature extraction on the calibration image to obtain a first feature map after processing; inputting the first feature map into a bounding box regression network to output a plurality of regions of interest corresponding to the calibration device in the first feature map through the bounding box regression network; generating a preset number of candidate boxes for each region of interest based on a preset prior box, each candidate box having a same size as the preset prior box, the preset number being determined according to a number of the plurality of patterns; inputting all candidate boxes and the first feature map into a center regression network to obtain coordinates of image center points of each pattern in the calibration image.

2. The image processing method of claim 1, wherein, The inputting all candidate boxes into the center regression network to determine the image center points of each pattern in the calibration image according to predicted image coordinates output by the center regression network comprises: inputting the first feature map and all candidate boxes into the center regression network to output a plurality of predicted image coordinates and a predicted probability corresponding to each predicted image coordinate according to the center regression network; arranging the plurality of predicted probabilities from high to low, and determining the predicted image coordinates corresponding to the preset number of predicted probabilities arranged in front as the first image coordinates of the image center points of each pattern in the calibration image.

3. The image processing method of claim 1, wherein, The plurality of patterns are a plurality of same and regularly arranged patterns, and the generating the preset number of candidate boxes for each region of interest based on the preset prior box comprises: obtaining a size of a circumscribed rectangle of any one pattern; generating the preset prior box according to the size of the circumscribed rectangle; for each region of interest, inputting the preset prior box and the region of interest into a region generation module to output the preset number of candidate boxes for each region of interest through the region generation module.

4. The image processing method of claim 3, wherein, The function expression of the region generation module is shown in the following formula (1): wherein, Proposal (k,n) refers to the nth candidate frame of the kth region of interest, bbox k refers to the kth region of interest, Base_anchor n refers to the nth candidate frame generated based on the preset prior frame, x1, x2 respectively refer to the horizontal coordinates of the diagonal points of the circumscribed rectangle corresponding to the preset prior frame, y1, y2 respectively refer to the vertical coordinates of the diagonal points of the circumscribed rectangle corresponding to the preset prior frame, x, y refer to the horizontal coordinates and vertical coordinates of the fixed point of each region of interest, w, h respectively refer to the width and height of each region of interest.

5. The image processing method of claim 1, wherein, The processing the calibration image to obtain the first feature map after processing comprises: inputting the calibration image into a residual network and outputting a plurality of second feature maps with different scales through the residual network; inputting the plurality of second feature maps into an up-sampling module and outputting a plurality of third feature maps with the same size through the up-sampling module; connecting all third feature maps to obtain a fourth feature map; inputting the fourth feature map into an attention mechanism module and outputting the first feature map after spatial dimension and channel dimension pooling processing of the fourth feature map through the attention mechanism module.

6. The image processing method of claim 1, further comprising: sorting actual center points of all patterns on the pattern board and obtaining relative positions between all sorted actual center points; determining world coordinates of each sorted actual center point according to the relative position corresponding to each sorted actual center point; performing perspective transformation on the first image coordinates of all image center points based on the relative positions between all sorted actual center points to determine the first image coordinates corresponding to each sorted actual center point. According to a corresponding relationship between the first image coordinates and the world coordinates of each sorted actual center point, the intrinsic parameters of the image acquisition device used to acquire the calibration image are determined according to all the first image coordinates and the world coordinates.

7. The image processing method of claim 6, wherein, The arrangement of the plurality of patterns is an array, the relative positions include arrangement intervals of the array, a number of rows of the array is a first number, a number of columns of the array is a second number, The sorting of the actual center points of all the patterns includes: The actual center points of all the patterns are sorted according to a row-first-column-second order of the array; The perspective transformation of the first image coordinates of all the image center points based on the relative positions of all the sorted actual center points includes: After the first image coordinates of each image center point are determined, two opposite corner points among all the image center points are determined according to all the first image coordinates; A plurality of first corner points of a first target polygon constructed in the calibration image are determined according to the two opposite corner points; A plurality of second corner points corresponding to the shape of the pattern plate are determined according to the arrangement intervals of the array, the first number and the second number, so as to construct a second target polygon corresponding to the shape of the pattern plate in the calibration image through the plurality of second corner points; Second image coordinates corresponding to each sorted actual center point are generated in the second target polygon based on the arrangement intervals of the array; A perspective transformation matrix between the first target polygon and the second target rectangle is determined according to the plurality of first corner points and the plurality of second corner points; Each first image coordinate is converted through the perspective transformation matrix to determine a second image coordinate corresponding to each first image coordinate, so as to determine the first image coordinates corresponding to each sorted actual center point.

8. The image processing method of claim 7, wherein, The determination of the plurality of first corner points of the first target polygon constructed in the calibration image according to the two opposite corner points includes: For any one of the two opposite corner points, a line between each center point and the opposite corner point is determined to determine that each line passes through the number of patterns; For each opposite corner point, the line passing through the number of patterns is the first target line, and the line passing through the number of patterns is the second target line; A line intersection point between all the first target lines and all the second target lines is determined; All the opposite corner points and all the line intersection points are determined as the plurality of first corner points of the first target polygon constructed in the calibration image.

9. An image processing apparatus, comprising: a memory configured to store instructions; and a processor configured to call the instructions from the memory and enable the image processing method according to any one of claims 1 to 8 when the instructions are executed.

10. An image processing system, comprising: an image acquisition device configured to acquire a calibration image of a calibration device; the calibration device comprising a pattern plate having a plurality of patterns; the image processing apparatus according to claim 9.

11. The image processing system of claim 10, wherein, The calibration device further comprises a heating plate connected to the pattern plate, so that each pattern in the calibration image presents obvious contrast with the background when the heating plate is heated, the image acquisition device comprises an infrared camera and a visible light camera, and the plurality of patterns are a plurality of hollow patterns which are the same and arranged in an array. 12.A machine readable storage medium having instructions stored thereon for causing a machine to perform the image processing method according to any one of claims 1 to 8.

Images