Method and device for acquiring same-semantic images of different sources, equipment and medium

Abstract

The invention discloses a method, a device and equipment for acquiring same-semantic images of different sources and a medium. The method comprises the following steps: receiving at least two images, respectively extracting feature points of a region of interest in the at least two images, taking any one of the at least two images as a reference image, matching the feature points in the image with the feature points in the reference image for each image in other images except the reference image, and obtaining a matching result; the method comprises the following steps: obtaining a matching feature point between an image and a reference image, calculating according to a preset transformation matrix calculation algorithm and the matching feature point to obtain a transformation matrix for converting the image into the reference image, converting the image through the transformation matrix to obtain a converted image, and taking the converted image as a same-semantic image of the reference image. Therefore, the same-semantic image of the reference image is obtained through the feature point matching and image conversion mode, compared with a cutting mode, the same-semantic image with high precision can be obtained, and the cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to a method, apparatus, device, and medium for acquiring semantically identical images from different sources. Background Technology

[0002] With the rapid development of image processing technology, the requirements for image processing are becoming increasingly higher. There is a need to obtain more accurate and reliable image data through image processing, such as obtaining more accurate semantic images from different sources. Here, different sources mean that the images come from different sources, specifically referring to different types of cameras used when acquiring image data.

[0003] The existing method for obtaining semantically similar images from different sources is to take multiple pictures of the same content using different types of cameras, and then manually crop the images to obtain semantically similar images from different sources.

[0004] It is evident that existing methods for obtaining semantically similar images from different sources mainly rely on manual image cropping, which results in high labor costs. Furthermore, the two semantically similar images obtained through cropping only appear similar to the naked eye; they are not identical at the pixel level, leading to low accuracy in the obtained semantically similar images. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for acquiring semantically identical images from different sources, which can acquire high-precision semantically identical images and reduce costs. The specific technical solution is as follows.

[0006] In a first aspect, the present invention provides a method for acquiring semantically identical images from different sources, applied to a processor in an image acquisition system. The image acquisition system further includes a physical reference frame and at least two cameras. The processor is communicatively connected to each camera. During use, the optical axes of each camera pass through the center of the physical reference frame. The at least two cameras, at the same position, respectively capture images of the object to be photographed, framed by the physical reference frame, obtaining at least two images, which are then sent to the processor. The regions of interest enclosed by the physical reference frames in the at least two images have the same semantic meaning. The at least two cameras are of different camera types, have the same resolution and field of view, and are located at the same position during shooting. The method includes:

[0007] Receive the at least two images;

[0008] Feature points of the regions of interest in the at least two images are extracted respectively;

[0009] Using any one of the at least two images as a reference image, for each of the other images besides the reference image, the feature points in the image are matched with the feature points in the reference image to obtain the matching feature points between the image and the reference image. A transformation matrix from the image to the reference image is calculated according to a preset transformation matrix calculation algorithm and the matching feature points. The image is then transformed using the transformation matrix to obtain the transformed image. The transformed image is used as a semantically related image to the reference image.

[0010] Optionally, the preset transformation matrix calculation algorithm is the least squares method.

[0011] Optionally, the step of transforming the image using the transformation matrix to obtain the transformed image includes:

[0012] The transformed image matrix is ​​obtained by multiplying the transformation matrix with the matrix composed of the pixels in the image.

[0013] Optionally, after the step of using the converted image as a semantically similar image of the reference image, the above-mentioned method for obtaining semantically similar images from different sources further includes:

[0014] A first image sample dataset is generated based on the reference image, and a second image sample dataset is generated based on the semantically similar images of the reference image.

[0015] The first image sample dataset and the second image sample dataset are used as training data to train the initial network model and obtain the target network model, wherein the target network model is used to associate the reference image with the corresponding semantic image.

[0016] Optionally, the step of generating a first image sample dataset based on the reference image and generating a second image sample dataset based on semantically similar images of the reference image includes:

[0017] The reference image is used as the first image, and any image from the semantically similar images of the reference image is selected as the second image.

[0018] A first image is obtained by performing an image geometric transformation on the first image, and a second image is obtained by performing the same image geometric transformation on the second image. This process is repeated multiple times to obtain multiple first images and multiple second images, wherein each image geometric transformation is different.

[0019] The first image and the plurality of transformed first images are used as the first image sample dataset, and the second image and the plurality of transformed second images are used as the second image sample dataset.

[0020] Secondly, the present invention provides a device for acquiring semantically identical images from different sources, applied to a processor in an image acquisition system. The image acquisition system further includes a physical reference frame and at least two cameras. The processor is communicatively connected to each camera. In use, the optical axes of each camera pass through the center of the physical reference frame. The at least two cameras, at the same position, respectively capture at least two images of the object to be photographed, framed by the physical reference frame, and send them to the processor. The regions of interest enclosed by the physical reference frames in the at least two images have the same semantic meaning. The at least two cameras have different camera types, the same resolution and field of view, and are located at the same position during shooting. The device includes:

[0021] A receiving module is configured to receive the at least two images;

[0022] The extraction module is used to extract feature points of the regions of interest in the at least two images respectively;

[0023] The conversion module is used to take any one of the at least two images as a reference image, and for each of the other images, match the feature points in the image with the feature points in the reference image to obtain the matching feature points between the image and the reference image. It calculates a transformation matrix from the image to the reference image according to a preset transformation matrix calculation algorithm and the matching feature points, and converts the image using the transformation matrix to obtain the converted image. The converted image is then used as a semantically related image of the reference image.

[0024] Optionally, the preset transformation matrix calculation algorithm is the least squares method.

[0025] Optionally, the conversion module is specifically used for:

[0026] The transformed image matrix is ​​obtained by multiplying the transformation matrix with the matrix composed of the pixels in the image.

[0027] Optionally, the aforementioned device for acquiring semantically related images from different sources further includes:

[0028] The generation module is configured to, after using the transformed image as a semantically similar image to the reference image, generate a first image sample dataset based on the reference image, and generate a second image sample dataset based on the semantically similar image to the reference image;

[0029] The model training module is used to train the initial network model using the first image sample dataset and the second image sample dataset as model training data to obtain the target network model, wherein the target network model is used to associate the reference image with the corresponding semantic image.

[0030] Optionally, the generation module includes:

[0031] The selection submodule is used to select the reference image as the first image and any one of the semantically related images of the reference image as the second image;

[0032] The transformation submodule is used to perform image geometric transformation on the first image to obtain a transformed first image, and to perform the same image geometric transformation on the second image to obtain a transformed second image. The steps of performing image geometric transformation on the first image to obtain a transformed first image and performing the same image geometric transformation on the second image to obtain a transformed second image are executed multiple times to obtain multiple transformed first images and multiple transformed second images, wherein each image geometric transformation is different.

[0033] A generation submodule is used to take the first image and the plurality of transformed first images as a first image sample dataset, and the second image and the plurality of transformed second images as a second image sample dataset.

[0034] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method for acquiring semantically identical images from different sources as provided in any embodiment of the present invention.

[0035] Fourthly, embodiments of the present invention provide an electronic device, the electronic device comprising:

[0036] One or more processors;

[0037] The processor is coupled to a storage device for storing one or more programs;

[0038] When the one or more programs are executed by the one or more processors, the electronic device implements the method for acquiring semantically related images from different sources as provided in any embodiment of the present invention.

[0039] As described above, the method, apparatus, device, and medium for acquiring semantically related images from different sources provided in this embodiment of the invention can receive at least two images, extract feature points of regions of interest from each of the at least two images, use any one of the at least two images as a reference image, and for each of the other images, match the feature points in that image with the feature points in the reference image to obtain matching feature points between the two images. A transformation matrix is ​​calculated from the image to the reference image based on a preset transformation matrix calculation algorithm and the matching feature points. The image is then transformed using the transformation matrix to obtain a transformed image, which is used as the semantically related image of the reference image. Therefore, by obtaining the semantically related image of the reference image through feature point matching and image transformation, compared to cropping, a higher accuracy semantically related image can be obtained, while reducing costs.

[0040] The innovative aspects of this invention include:

[0041] 1. By using feature point matching and image transformation, a semantically equivalent image of the reference image can be obtained. Compared with cropping, this method can obtain a more accurate semantically equivalent image and reduce costs.

[0042] 2. By accurately calibrating the camera's intrinsic and extrinsic parameters, the correspondence between semantic images can be estimated more precisely, thereby improving the accuracy and consistency of image alignment.

[0043] 3. Always use a camera stabilization device to fix the camera. This can reduce the problem of large differences in the captured images due to camera shaking or movement, thereby improving the accuracy of image alignment.

[0044] 4. The method proposed in this invention has wide applicability and is suitable for various source images, including images taken by different camera models under different lighting conditions and environmental backgrounds. Whether using a professional camera, a consumer camera, or a research-grade camera, and whether shooting indoors or outdoors, this method can be used to obtain high-precision semantically meaningful images.

[0045] 5. This invention provides a simple and effective method for acquiring semantically identical images from different sources. By selecting a suitable camera and setting an entity reference box, it reduces the cost and complexity of acquiring semantically identical images from different sources without the need for complex equipment or technology.

[0046] 6. By using a camera fixing device to stabilize the camera and calibrating the camera's intrinsic and extrinsic parameters, the problem of large differences in the captured images caused by camera shake or movement is reduced, thereby improving the accuracy of image alignment. Furthermore, by accurately calibrating the camera's intrinsic and extrinsic parameters, the correspondence between semantic images can be estimated more precisely, thereby improving the accuracy and consistency of image alignment and further enhancing the quality and accuracy of the acquired semantic images.

[0047] 7. By performing image geometric transformations on the first and second images, more training samples are generated, thereby increasing the diversity and generalization ability of the training data. This further enhances the robustness of the model to different transformations and perturbations, and improves the accuracy and quality of semantically related images.

[0048] 8. By training the initial network model using the above training method, a target network model can be obtained to associate the reference image with the corresponding semantically related image. This target network model can then be used to obtain the semantically related image. Since the image sample dataset used during model training mainly consists of high-precision semantically related images, the performance of the trained model is improved. This allows the trained target network model to accurately convert and process images, thereby obtaining high-precision semantically related images.

[0049] 9. The high-precision semantic images obtained can be used for image processing tasks, providing a better data foundation for image processing tasks and having broad application prospects.

[0050] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0052] Figure 1(a) is a schematic diagram of an image captured by camera A;

[0053] Figure 1(b) is a schematic diagram of an image captured by camera B;

[0054] Figure 2 This is a flowchart illustrating a method for acquiring semantically related images from different sources, as provided in an embodiment of the present invention.

[0055] Figure 3(a) is a schematic diagram of an image captured by camera A;

[0056] Figure 3(b) is a schematic diagram of the image taken by camera B;

[0057] Figure 4 This invention provides an apparatus for acquiring semantically related images from different sources.

[0058] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0059] Figure 1(a)-Figure 5 In the diagram, 1 is a physical reference frame, 2 is an optical axis, 3 is the object to be photographed, 4 is camera A, and 5 is camera B. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0061] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0062] This invention discloses a method, apparatus, device, and medium for acquiring semantically similar images from different sources, which can acquire highly accurate semantically similar images and reduce costs. The embodiments of this invention are described in detail below.

[0063] Figure 1(a) is a schematic diagram of an image captured by camera A, and Figure 1(b) is a schematic diagram of an image captured by camera B.

[0064] See Figure 1(a) and 1(b) This method is applied to a processor in an image acquisition system, which also includes a physical reference frame 1 and at least two cameras, with the processor communicatively connected to each camera. Figure 1(a) and 1(b) The horizontal dashed line represents a ray of light.

[0065] In this case, at least two cameras are of different types, resulting in images from different sources. For example, one might be an IR image captured by a TOF camera, and the other a grayscale image captured by an RGB camera.

[0066] TOF camera: Time-of-flight camera, a type of camera that obtains image depth information by measuring the round-trip time of light. TOF cameras also record the intensity information of infrared light and convert it into grayscale values ​​to form an infrared image, or IR (Infrared Radiation) image.

[0067] RGB cameras: These typically contain image sensors with three channels: red, green, and blue, used to capture color information; hence, they are color cameras. "R" stands for Red, "G" for Green, and "B" for Blue. They can also be used to acquire grayscale images. In grayscale mode, a color camera can obtain grayscale images by either weighted averaging of the pixel values ​​from the three channels or by selecting the pixel value from one channel.

[0068] To improve the accuracy of the acquired semantically identical images, at least two cameras can be high-resolution cameras, and at least two cameras are set to have the same resolution and both are preset high resolutions to obtain high-quality images. At least two cameras are also set to have the same field of view and are located in the same position when shooting, so that at least two cameras can capture the same image content. Here, high resolution refers to the characteristics of a large number of pixels, fine image quality, and high clarity contained in an image, video, or display device.

[0069] To further improve the accuracy and consistency of semantically related images, camera intrinsic and extrinsic parameters can be calibrated. The camera's intrinsic parameters include focal length, principal point position, and distortion parameters, while its extrinsic parameters include camera position and pose information. Since acquiring semantically related images is essentially image alignment, accurately calibrating the camera's intrinsic and extrinsic parameters allows for a more precise estimation of the correspondence between semantically related images, thereby improving the accuracy and consistency of image alignment.

[0070] To ensure the stability and consistency of the camera position, a camera stabilization device, such as a tripod or stabilizer, is used to hold the camera in place throughout the shooting process. This reduces the problem of significant differences in the captured images due to camera shake or movement, thereby improving the accuracy of image alignment.

[0071] See also Figure 1(a) and 1(b)The physical reference frame 1 can be a rectangular reference frame. During the shooting process, a reference frame fixing device, such as a tripod or stabilizing bracket, is used to fix the physical reference frame 1. The function of the physical reference frame 1 is to determine the region of interest in the image. In machine vision and image processing, the region of interest is the area to be processed delineated from the image being processed using rectangles, circles, ellipses, irregular polygons, etc. In this embodiment of the invention, the region of interest is the area enclosed by the physical reference frame in the image.

[0072] In use, the optical axis 2 of each camera passes through the center of the physical reference frame 1. At least two cameras, positioned at the same location, capture images of the object 3 to be photographed, framed by the physical reference frame 1, obtaining at least two images, which are then sent to the processor. The object 3 to be photographed can be an object or a scene, for example... Figure 1(a) and 1(b) The upward arrows in the image represent objects, but the arrows themselves have no practical meaning.

[0073] Specifically, during the shooting process, one of the cameras can be selected as the reference camera. For example, if camera A4 is selected as the reference camera, see Figure 1(a). At this time, the region of interest selected by the entity reference frame 1 is the area enclosed by A1, B1, C1 and D1. The other cameras are based on the reference camera, and the regions of interest selected by the corresponding entity reference frame 1 are also the same areas. This can be achieved by adjusting the magnification of the other cameras.

[0074] Continuing with 1(b), for example: the other camera is camera B5, and the region of interest selected by the entity reference frame 1 is the area enclosed by A2, B2, C2 and D2. The regions of interest of the two cameras are exactly the same. The purpose of this is to ensure that at least two cameras completely capture the object to be photographed 3 enclosed by the entity reference frame 1 during shooting, so that the image semantics of the region of interest enclosed by the entity reference frame are the same in at least two images.

[0075] During the shooting process, it is necessary to keep the horizontal and vertical alignment of each camera to avoid image rotation or distortion.

[0076] Figure 2 This is a flowchart illustrating a method for acquiring semantically related images from different sources, as provided in an embodiment of the present invention. The method specifically includes the following steps.

[0077] S110: Receive at least two images.

[0078] The processor in the image acquisition system receives images from each camera, meaning it receives at least two images. In other words, the processor receives as many images as there are cameras. Each image captures the same content.

[0079] Figure 3(a) is a schematic diagram of an image taken by camera A, and Figure 3(b) is a schematic diagram of an image taken by camera B. For ease of understanding, the following uses camera A and camera B as examples to introduce the contents of the images taken by the two cameras. Refer to Figures 3(a) and 3(b). Assuming that the outer perimeter of the captured image is parallel to the outer perimeter of the region of interest, it can be seen that the image taken by camera A not only contains the region of interest enclosed by A1, B1, C1, and D1, but also contains other content outside the region of interest enclosed by A1, B1, C1, and D1. Similarly, the image taken by camera B not only contains the region of interest enclosed by A2, B2, C2, and D2, but also contains other content outside the region of interest enclosed by A2, B2, C2, and D2. The distance between A1 and C1 and the left side of the image taken by camera A is d1, the distance between B1 and D1 and the right side of the image taken by camera A is d2, the distance between A2 and C2 and the left side of the image taken by camera B is d3, and the distance between B2 and D2 and the right side of the image taken by camera B is d4.

[0080] Since d1 is less than d3 and d2 is greater than d4, although the regions of interest enclosed by entity reference boxes in the two images have the same semantic meaning, the two images are not aligned and are not images with the same semantic meaning.

[0081] S120: Extract feature points of the region of interest from at least two images respectively.

[0082] To obtain images with the same semantic meaning, at least two images need to be aligned. Specifically, feature points of the regions of interest in at least two images need to be extracted.

[0083] Extracting feature points from at least two images may include:

[0084] Feature points are extracted from at least two images using a preset feature point extraction algorithm.

[0085] The preset feature point extraction algorithm can be either SIFT (Scale-invariant feature transform) or ORB (Oriented Fast and Rotated Brief) algorithm.

[0086] The extracted feature points are points with features in the image, such as the corner points of a building or the corner points of a region of interest.

[0087] For ease of understanding, let's take the images taken by camera A and camera B as examples. Assume that the extracted feature points are the corner points of the region of interest. Then the feature points in the image taken by camera A are (x1,y1), (x2,y2), (x3,y3), (x4,y4), and the feature points in the image taken by camera B are (x'1,y'1), (x'2,y'2), (x'3,y'3), (x'4,y'4).

[0088] S130: Take any one of at least two images as a reference image. For each image other than the reference image, match the feature points in the image with the feature points in the reference image to obtain the matching feature points between the image and the reference image. Calculate the transformation matrix from the image to the reference image according to the preset transformation matrix calculation algorithm and the matching feature points. Transform the image using the transformation matrix to obtain the transformed image. Use the transformed image as a semantically related image to the reference image.

[0089] To align at least two images, after extracting feature points, it is necessary to match the feature points, calculate the transformation matrix, and apply the transformation matrix to obtain images with the same semantic meaning.

[0090] Since there may be more than two images, one image can be used as a reference image during alignment, and then the reference image can be aligned with the other images respectively.

[0091] Specifically, any one of at least two images is used as a reference image. For each of the other images, the feature points in the reference image are matched with the feature points in the reference image to obtain the matching feature points between the reference image and the reference image.

[0092] The method for matching feature points in the image with feature points in the reference image can be to use a preset feature point matching algorithm. For example, the preset feature point matching algorithm can be the nearest neighbor matching algorithm or the RANSAC (Random Sample Consensus) algorithm.

[0093] Continuing with the example of images captured by camera A and camera B, following the example of step S120, assume that there are 4 pairs of matching feature points: (x1, y1) and (x'1, y'1), (x2, y2) and (x'2, y'2), (x3, y3) and (x'3, y'3), (x4, y4) and (x'4, y'4).

[0094] After obtaining the matching feature points, the transformation matrix from the image to the reference image is calculated based on the preset transformation matrix calculation algorithm and the matching feature points. The preset transformation matrix calculation algorithm can be the least squares method.

[0095] Continuing with the example above, assuming the image captured by camera A is the reference image and the transformation matrix is ​​M, then the transformation matrix between the image captured by camera A and the image captured by camera B, obtained using the least squares method, is:

[0096] M = (P2) T *P1)*(P1 T *P1) -1

[0097] Where T represents the transpose of the matrix, -1 represents the inverse of the matrix, P1 is the matrix composed of matching feature points in the image taken by camera A, and P2 is the matrix composed of matching feature points in the image taken by camera B.

[0098] After obtaining the transformation matrix, the image is transformed using the transformation matrix to obtain the transformed image, which is geometrically aligned with the reference image, thus becoming semantically identical images. The transformed image is then used as the semantically identical image of the reference image. In other words, the resulting transformed image and the reference image not only have the same semantic meaning in the region of interest, but also in the entire image. Therefore, a high-precision semantically identical image is obtained.

[0099] The transformation of the image using a transformation matrix to obtain the transformed image can include:

[0100] The transformed image matrix is ​​obtained by multiplying the transformation matrix with the matrix composed of the pixels in the image.

[0101] Continuing with the example above, assuming the matrix of pixels in the image captured by camera B is I2, then multiplying the transformation matrix M by the matrix I2 of pixels in the image captured by camera B yields the image matrix I2' of the transformed image.

[0102] I2'=M*I2

[0103] The resulting transformed image is aligned with the reference image, and the content of the two images is identical. Furthermore, for every other image besides the reference image, there will be a corresponding semantic image.

[0104] As described above, this embodiment of the invention can receive at least two images, extract feature points of regions of interest from each of the at least two images, use any one of the at least two images as a reference image, and for each of the other images, match the feature points in that image with the feature points in the reference image to obtain matching feature points between the two images. A transformation matrix is ​​calculated from the image to the reference image based on a preset transformation matrix calculation algorithm and the matching feature points. The image is then transformed using the transformation matrix to obtain a transformed image, which is used as a semantically equivalent image of the reference image. Therefore, by obtaining a semantically equivalent image of the reference image through feature point matching and image transformation, compared to cropping, a more accurate semantically equivalent image can be obtained, while reducing costs.

[0105] Furthermore, the method proposed in this invention has wide applicability, applicable to various source images, including images taken by different camera models under different lighting conditions and environmental backgrounds. Whether using a professional camera, a consumer camera, or a research-grade camera, and whether shooting indoors or outdoors, this method can be used to obtain high-precision semantically meaningful images.

[0106] Furthermore, this invention provides a simple and effective method for acquiring semantically identical images from different sources. By selecting a suitable camera and setting an entity reference box, it reduces the cost and complexity of acquiring semantically identical images from different sources without the need for complex equipment or technology.

[0107] Furthermore, by using a camera fixing device to stabilize the camera and calibrating the camera's intrinsic and extrinsic parameters, the problem of significant differences in the captured images caused by camera shake or movement is reduced, thereby improving the accuracy of image alignment. In addition, by accurately calibrating the camera's intrinsic and extrinsic parameters, the correspondence between semantic images can be estimated more precisely, thereby improving the accuracy and consistency of image alignment, and further improving the quality and accuracy of the acquired semantic images.

[0108] Following step S130 above, the method for obtaining semantically related images from different sources provided in this embodiment of the invention may further include:

[0109] First image sample data is generated based on a reference image, and second image sample data is generated based on a semantically similar image to the reference image.

[0110] The first image sample data and the second image sample data are used as training data to train the initial network model and obtain the target network model. The target network model is used to associate the reference image with the corresponding semantic image.

[0111] After obtaining the semantically equivalent image of the reference image, the reference image and the corresponding semantically equivalent image can be used for image processing tasks. The most common image processing task is pixel-to-pixel cross-domain transformation training and application.

[0112] Specifically, a first image sample dataset is generated based on a reference image, and a second image sample dataset is generated based on semantically similar images of the reference image;

[0113] The first image sample dataset and the second image sample dataset are used as training data to train the initial network model and obtain the target network model. The target network model is used to associate the reference image with the corresponding semantic image.

[0114] The process of generating a first image sample dataset based on a reference image and generating a second image sample dataset based on semantically related images of the reference image may include:

[0115] Use the reference image as the first image, and select any image from the semantically similar images of the reference image as the second image;

[0116] A first image is transformed by performing an image geometric transformation on the first image to obtain a transformed first image. A second image is transformed by performing the same image geometric transformation on the first image to obtain a transformed second image. This process is repeated multiple times to obtain multiple transformed first images and multiple transformed second images, wherein each image geometric transformation is different.

[0117] The first image and multiple transformed first images are used as the first image sample dataset, and the second image and multiple transformed second images are used as the second image sample dataset.

[0118] Since there may be multiple semantically related images of the reference image, each of which can form a semantically related image pair with the reference image, but only one pair of semantically related images is needed each time during training, the reference image needs to be used as the first image, and any one of the semantically related images of the reference image needs to be selected as the second image.

[0119] To increase the diversity and generalization ability of training data, data augmentation and enlargement methods can be used to increase the amount of data. For example, geometric transformations can be performed on the first image to generate more training samples, and the corresponding second image also needs to undergo similar geometric transformations. Geometric transformations involve spatial geometric transformations of image pixels without altering the image content, and mainly include translation, scaling, rotation, flipping, and mirroring.

[0120] Specifically, a first image is subjected to an image geometric transformation to obtain a transformed first image, and a second image is subjected to the same image geometric transformation as the first image to obtain a transformed second image. This process is repeated multiple times to obtain multiple transformed first images and multiple transformed second images, wherein each image geometric transformation is different.

[0121] Then, the first image and multiple transformed first images are used as the first image sample dataset, and the second image and multiple transformed second images are used as the second image sample dataset.

[0122] Therefore, by performing image geometric transformations on the first and second images, more training samples are generated, thereby increasing the diversity and generalization ability of the training data, further increasing the robustness of the model to different transformations and perturbations, and improving the accuracy and quality of semantically related images.

[0123] After obtaining the first image sample dataset and the second image sample dataset, the first image sample dataset and the second image sample dataset can be used as model training data to train the initial network model and obtain the target network model. The target network model is used to associate the reference image with the corresponding semantic image.

[0124] Specifically, the first image sample dataset and the second image sample dataset are used as training data to train the initial network model, resulting in the target network model, which may include:

[0125] The first image sample dataset and the second image sample dataset are input into the initial network model, which includes a feature extraction layer and a regression layer.

[0126] For each image in the first image sample dataset, the feature vector corresponding to the image is determined by the first model parameters of the feature extraction layer. The feature vector is then regressed by the second model parameters of the regression layer to obtain an initial semantically identical image. The difference between the initial semantically identical image and the third image is calculated, where the third image is the semantically identical image corresponding to the image in the second image sample data.

[0127] Adjust the parameters of the first and second models based on the difference values;

[0128] When the number of iterations reaches the preset number, the training is completed, and a target network model that associates the reference image with the corresponding semantic image is obtained.

[0129] Understandably, the first step is to build an initial network model, then train it to obtain the target network model. In one implementation, the Caffe tool can be used to build an initial network model that includes feature extraction layers and regression layers. For example, the initial network model can be an SVM (Support Vector Machine), an RNN (Recurrent Neural Network), an LSTM (Long Short-Term Memory), or a GRU (Gated Recurrent Unit), etc.

[0130] After constructing the initial network model, the first image sample dataset and the second image sample dataset are input into the initial network model for training.

[0131] Specifically, the first image sample dataset and the second image sample dataset are input into the feature extraction layer. For each image in the first image sample dataset, the feature vector corresponding to the image is determined by the first model parameters of the feature extraction layer. The feature vector is then regressed by the second model parameters of the regression layer to obtain the initial semantically identical image.

[0132] Then, the difference between the initial semantically identical image and the third image is calculated, where the third image is the semantically identical image corresponding to the image in the second image sample data.

[0133] After obtaining the difference values, the parameters of the first and second models can be adjusted based on the difference values. Specifically, the parameters of the first and second models can be adjusted based on the difference values ​​through the evaluation function in the backpropagation method.

[0134] When the number of iterations reaches the preset number, the training is completed, and a target network model that associates the reference image with the corresponding semantic image is obtained.

[0135] As can be seen, training the initial network model using the above training method yields a target network model that associates a reference image with its corresponding semantically related image. This target network model can then be used to obtain the semantically related image. Since the image sample dataset used during model training primarily consists of high-precision semantically related images, the performance of the trained model is improved. This allows the trained target network model to accurately convert and process images, thereby obtaining high-precision semantically related images.

[0136] The target network model can be a super-resolution image reconstruction model, a deblurring model, or a style transfer model.

[0137] It is evident that the acquired high-precision semantically related images can be used for image processing tasks, providing a better data foundation for image processing tasks and showing broad application prospects.

[0138] Figure 4 This invention provides a device for acquiring semantically identical images from different sources, applied to a processor in an image acquisition system. The image acquisition system further includes a physical reference frame and at least two cameras. The processor is communicatively connected to each camera. During use, the optical axes of each camera pass through the center of the physical reference frame. The at least two cameras, at the same position, capture at least two images of the object to be photographed, framed by the physical reference frame, and send them to the processor. The regions of interest enclosed by the physical reference frames in the at least two images have the same semantic meaning. The at least two cameras are of different camera types, have the same resolution and field of view, and are located at the same position during shooting. See [link to relevant documentation]. Figure 4 The device includes:

[0139] Receiver module 410 is used to receive the at least two images;

[0140] Extraction module 420 is used to extract feature points of the regions of interest in the at least two images respectively;

[0141] The conversion module 430 is used to take any one of the at least two images as a reference image, and for each of the other images, match the feature points in the image with the feature points in the reference image to obtain the matching feature points between the image and the reference image. It calculates a transformation matrix from the image to the reference image according to a preset transformation matrix calculation algorithm and the matching feature points, and converts the image using the transformation matrix to obtain the converted image. The converted image is then used as a semantically related image of the reference image.

[0142] The apparatus provided in this invention can receive at least two images, extract feature points of regions of interest from each of the at least two images, use any one of the at least two images as a reference image, and for each of the other images, match the feature points in the reference image with the feature points in the other image to obtain matching feature points between the two images. A transformation matrix is ​​calculated from the image to the reference image based on a preset transformation matrix calculation algorithm and the matching feature points. The image is then transformed using the transformation matrix to obtain a transformed image, which is used as a semantically equivalent image of the reference image. Therefore, by obtaining a semantically equivalent image of the reference image through feature point matching and image transformation, a higher-precision semantically equivalent image can be obtained compared to cropping, while reducing costs.

[0143] In one implementation, the preset transformation matrix calculation algorithm is the least squares method.

[0144] In one implementation, the conversion module 430 can be specifically used for:

[0145] The transformed image matrix is ​​obtained by multiplying the transformation matrix with the matrix composed of the pixels in the image.

[0146] In one implementation, the apparatus may further include:

[0147] The generation module is configured to, after using the transformed image as a semantically similar image to the reference image, generate a first image sample dataset based on the reference image, and generate a second image sample dataset based on the semantically similar image to the reference image;

[0148] The model training module is used to train the initial network model using the first image sample dataset and the second image sample dataset as model training data to obtain the target network model, wherein the target network model is used to associate the reference image with the corresponding semantic image.

[0149] In one implementation, the generation module may include:

[0150] The selection submodule is used to select the reference image as the first image and any one of the semantically related images of the reference image as the second image;

[0151] The transformation submodule is used to perform image geometric transformation on the first image to obtain a transformed first image, and to perform the same image geometric transformation on the second image to obtain a transformed second image. The steps of performing image geometric transformation on the first image to obtain a transformed first image and performing the same image geometric transformation on the second image to obtain a transformed second image are executed multiple times to obtain multiple transformed first images and multiple transformed second images, wherein each image geometric transformation is different.

[0152] A generation submodule is used to take the first image and the plurality of transformed first images as a first image sample dataset, and the second image and the plurality of transformed second images as a second image sample dataset.

[0153] The above-described apparatus embodiments correspond to the method embodiments and have the same technical effects. For detailed explanations, please refer to the method embodiments. The apparatus embodiments are derived from the method embodiments; detailed explanations can be found in the method embodiments section, and will not be repeated here.

[0154] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, the electronic device includes:

[0155] One or more processors 510;

[0156] The processor 510 is coupled to a storage device 520, the storage device 520 being used to store one or more programs;

[0157] When the one or more programs are executed by the one or more processors 510, the electronic device implements the method for acquiring semantically related images from different sources as provided in any embodiment of the present invention.

[0158] Based on the above method embodiments, another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for acquiring semantically identical images from different sources as provided in any embodiment of the present invention.

[0159] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0160] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for acquiring semantically identical images from different sources, characterized in that, A processor is applied to an image acquisition system, the image acquisition system further including a physical reference frame and at least two cameras. The processor is communicatively connected to each camera. In use, the optical axes of each camera pass through the center of the physical reference frame. The at least two cameras, at the same position, respectively capture at least two images of the object to be photographed, which are framed by the physical reference frame, and send them to the processor. The regions of interest enclosed by the physical reference frame in the at least two images have the same semantic meaning. The at least two cameras have different camera types, the same resolution and field of view, and are located at the same position during shooting. The method includes: Receive the at least two images; Feature points of the regions of interest in the at least two images are extracted respectively; Using any one of the at least two images as a reference image, for each of the other images besides the reference image, the feature points in the image are matched with the feature points in the reference image to obtain the matching feature points between the image and the reference image. A transformation matrix from the image to the reference image is calculated according to a preset transformation matrix calculation algorithm and the matching feature points. The image is then transformed using the transformation matrix to obtain the transformed image. The transformed image is used as a semantically related image to the reference image.

2. The method as described in claim 1, characterized in that, The preset transformation matrix calculation algorithm is the least squares method.

3. The method as described in claim 1 or 2, characterized in that, The step of transforming the image using the transformation matrix to obtain the transformed image includes: The transformed image matrix is ​​obtained by multiplying the transformation matrix with the matrix composed of the pixels in the image.

4. The method as described in claim 1, characterized in that, After the step of using the transformed image as a semantically similar image to the reference image, the method further includes: A first image sample dataset is generated based on the reference image, and a second image sample dataset is generated based on the semantically similar images of the reference image. The first image sample dataset and the second image sample dataset are used as training data to train the initial network model and obtain the target network model, wherein the target network model is used to associate the reference image with the corresponding semantic image.

5. The method as described in claim 4, characterized in that, The steps of generating a first image sample dataset based on the reference image and generating a second image sample dataset based on semantically related images of the reference image include: The reference image is used as the first image, and any image from the semantically similar images of the reference image is selected as the second image. A first image is obtained by performing an image geometric transformation on the first image, and a second image is obtained by performing the same image geometric transformation on the second image. This process is repeated multiple times to obtain multiple first images and multiple second images, wherein each image geometric transformation is different. The first image and the plurality of transformed first images are used as the first image sample dataset, and the second image and the plurality of transformed second images are used as the second image sample dataset.

6. A device for acquiring semantically related images from different sources, characterized in that, A processor is applied to an image acquisition system, the image acquisition system further including a physical reference frame and at least two cameras. The processor is communicatively connected to each camera. In use, the optical axis of each camera passes through the center of the physical reference frame. The at least two cameras, at the same position, respectively capture at least two images of the object to be photographed, which are framed by the physical reference frame, and send them to the processor. The regions of interest enclosed by the physical reference frame in the at least two images have the same semantic meaning. The at least two cameras have different camera types, the same resolution and field of view, and are located at the same position during shooting. The device includes: A receiving module is configured to receive the at least two images; The extraction module is used to extract feature points of the regions of interest in the at least two images respectively; The conversion module is used to take any one of the at least two images as a reference image, and for each of the other images, match the feature points in the image with the feature points in the reference image to obtain the matching feature points between the image and the reference image. It calculates a transformation matrix from the image to the reference image according to a preset transformation matrix calculation algorithm and the matching feature points, and converts the image using the transformation matrix to obtain the converted image. The converted image is then used as a semantically related image of the reference image.

7. The apparatus as claimed in claim 6, characterized in that, The preset transformation matrix calculation algorithm is the least squares method.

8. The apparatus as claimed in claim 6 or 7, characterized in that, The conversion module is specifically used for: The transformed image matrix is ​​obtained by multiplying the transformation matrix with the matrix composed of the pixels in the image.

9. The apparatus as claimed in claim 6, characterized in that, The device further includes: The generation module is configured to, after using the transformed image as a semantically similar image to the reference image, generate a first image sample dataset based on the reference image, and generate a second image sample dataset based on the semantically similar image to the reference image; The model training module is used to train the initial network model using the first image sample dataset and the second image sample dataset as model training data to obtain the target network model, wherein the target network model is used to associate the reference image with the corresponding semantic image.

10. The apparatus as claimed in claim 9, characterized in that, The generation module includes: The selection submodule is used to select the reference image as the first image and any one of the semantically related images of the reference image as the second image; The transformation submodule is used to perform image geometric transformation on the first image to obtain a transformed first image, and to perform the same image geometric transformation on the second image to obtain a transformed second image. The steps of performing image geometric transformation on the first image to obtain a transformed first image and performing the same image geometric transformation on the second image to obtain a transformed second image are executed multiple times to obtain multiple transformed first images and multiple transformed second images, wherein each image geometric transformation is different. A generation submodule is used to take the first image and the plurality of transformed first images as a first image sample dataset, and the second image and the plurality of transformed second images as a second image sample dataset.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

12. An electronic device, characterized in that, The electronic device includes: One or more processors; The processor is coupled to a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the electronic device performs the method as described in any one of claims 1-5.

Images