Camera calibration method and system for cooking device

Abstract

The application relates to the field of cooking equipment and provides a camera calibration method and system of cooking equipment, which comprises the following steps: inputting a target image of a target cooking range into a target recognition model to obtain first coordinates of multiple target objects in the target cooking range output by the target recognition model; performing cluster analysis on the first coordinates to obtain second coordinates of the multiple target objects; acquiring spatial position features of the multiple target objects in the target cooking range; and obtaining camera calibration parameters of a camera of the cooking equipment based on the second coordinates and the spatial position features; wherein the target image is collected by the camera, and the target recognition model is trained by taking a sample image as a sample and taking coordinates of a sample object corresponding to the sample image as a sample label. The method combines deep learning and cluster analysis to calculate camera calibration parameters, calibrate the camera, and improve the recognition accuracy of the camera.

Description

Technical Field

[0001] This invention relates to the field of cooking equipment technology, and in particular to a method and system for calibrating cameras in cooking equipment. Background Technology

[0002] As living standards improve, people have higher and higher requirements for food, and more and more people will use their spare time to cook to satisfy their love of food.

[0003] In existing technology, cameras on cooking equipment can capture and identify the items and actions being cooked to assist users. However, due to significant differences in the installation environment of cooking equipment, the cameras may not capture images as accurately as they were originally set to, thus reducing the accuracy of visual recognition. This prevents the cooking equipment from precisely assisting users in cooking, affecting their experience and reducing the enjoyment of cooking. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a camera calibration method for cooking equipment to improve the accuracy of camera recognition.

[0005] A camera calibration method for a cooking apparatus according to a first aspect of the present invention includes:

[0006] The target image of the target stove is input into the target recognition model to obtain the first coordinates of multiple target objects in the target stove output by the target recognition model;

[0007] Perform cluster analysis on the first coordinates to obtain the second coordinates of the multiple target objects;

[0008] Obtain the spatial position features of the multiple target objects in the target stove;

[0009] Based on the second coordinates and the spatial location features, the camera calibration parameters of the cooking device's camera are obtained;

[0010] The target image is acquired by the camera, and the target recognition model is trained using sample images as samples and pre-determined coordinates of sample objects corresponding to the sample images as sample labels.

[0011] According to the camera calibration method of the cooking device of the present invention, deep learning is used to identify target objects on the target stove, cluster analysis is used to improve the accuracy of the coordinates of the target objects, and the camera calibration parameters of the camera are determined based on the second coordinates and spatial position characteristics of the target objects. This helps to obtain more accurate images of the target stove, improve the accuracy of subsequent visual tasks, accurately identify information such as pots, stoves, ingredients and cooking behavior on the target stove, assist in cooking, and enhance the fun of cooking.

[0012] According to one embodiment of the present invention, obtaining the camera calibration parameters of the camera of the cooking device based on the second coordinates and the spatial position features includes:

[0013] Based on the second coordinates and the spatial location features, multiple sets of equations are obtained regarding the camera calibration parameters;

[0014] The camera calibration parameters are obtained by performing matrix transformations on multiple sets of equations and using the least squares method.

[0015] According to one embodiment of the present invention, the camera calibration parameters include intrinsic and extrinsic parameters, and the equation is:

[0016]

[0017] in, The second coordinate is... For the aforementioned spatial location features, Let [R|t] be the intrinsic parameter, [R|t] be the extrinsic parameter, R and t be homogeneous transformation matrices, x0 and y0 be the origin of the second coordinates, and α and s be the distortion parameters of the intrinsic parameter.

[0018] The equation obtained after matrix transformation is:

[0019]

[0020] The second coordinate is... For the aforementioned spatial location features, The camera is calibrated.

[0021] According to an embodiment of the present invention, the step of performing cluster analysis on the first coordinates to obtain the second coordinates of the plurality of target objects includes:

[0022] The number of the multiple target objects is obtained, and the number of the multiple target objects is used as the number of clusters for cluster analysis;

[0023] Based on the number of clusters, the first coordinates are subjected to cluster analysis to obtain the second coordinates.

[0024] According to one embodiment of the present invention, obtaining the number of the plurality of target objects includes:

[0025] Obtain the device information of the target stove;

[0026] Based on the device information, the number of the plurality of target objects is obtained.

[0027] According to one embodiment of the present invention, obtaining the number of the plurality of target objects includes:

[0028] The target image of the target stove is input into the target recognition model to obtain the number of the multiple target objects in the target stove as output by the target recognition model.

[0029] According to an embodiment of the present invention, the step of performing cluster analysis on the first coordinates based on the number of clusters to obtain the second coordinates includes:

[0030] Based on the number of clusters, the first coordinates are divided into the first coordinates of different target objects;

[0031] By obtaining the first coordinates of two diagonal points in the first coordinates of different target objects, the coordinates of the center point of the target object are obtained;

[0032] Cluster analysis is performed on the coordinates of the center point to obtain the coordinates of the target center point;

[0033] Calculate the distance between each point in the first coordinate system of the target object and the coordinate system of the center point of the target object;

[0034] The distance value is compared with a distance threshold, and points whose distance value is greater than the distance threshold are discarded.

[0035] The points whose distance values ​​are not greater than the distance threshold are retained to obtain the second coordinates.

[0036] A camera calibration system for a cooking apparatus according to a second aspect embodiment of the present invention includes:

[0037] Cooking equipment;

[0038] A camera is mounted on the cooking device;

[0039] The target stove is equipped with multiple target objects and is located within the shooting area of ​​the camera.

[0040] A controller, electrically connected to the camera, is configured to control the camera to perform calibration based on the aforementioned camera calibration method for cooking equipment.

[0041] A camera calibration device for a cooking apparatus according to a third aspect embodiment of the present invention includes:

[0042] The first processing module is used to input the target image of the target stove into the target recognition model and obtain the first coordinates of multiple target objects in the target stove output by the target recognition model;

[0043] The second processing module is used to perform cluster analysis on the first coordinates to obtain the second coordinates of the multiple target objects;

[0044] The acquisition module is used to acquire the spatial position features of the multiple target objects in the target stove;

[0045] The calculation module is used to obtain the camera calibration parameters of the camera of the cooking device based on the second coordinates and the spatial position features;

[0046] The target image is acquired by the camera, and the target recognition model is trained using sample images as samples and pre-determined coordinates of sample objects corresponding to the sample images as sample labels.

[0047] An electronic device according to a fourth aspect of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the camera calibration method of any of the above-described cooking devices.

[0048] According to a fifth aspect of the present invention, a non-transitory computer-readable storage medium is provided thereon storing a computer program that, when executed by a processor, implements the steps of the camera calibration method for any of the above-described cooking devices.

[0049] A computer program product according to a sixth aspect of the present invention includes a computer program that, when executed by a processor, implements the steps of the camera calibration method for any of the cooking devices described above.

[0050] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0051] Determining the camera calibration parameters based on the second coordinates and spatial location characteristics of the target object helps to obtain more accurate images of the target stove, improves the accuracy of subsequent visual tasks, and accurately identifies information such as pots, stoves, ingredients, and cooking behaviors on the target stove, thus assisting in cooking and enhancing the enjoyment of cooking.

[0052] Furthermore, by using the number of target objects on the target stove as the cluster number, cluster analysis can effectively distinguish the first coordinates of different target objects, thereby improving the accuracy of the obtained second coordinates.

[0053] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic flowchart of the camera calibration method for cooking equipment provided in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the structure of the camera in the cooking device provided in an embodiment of the present invention;

[0057] Figure 3 This is a coordinate transformation diagram of the camera calibration method for cooking equipment provided in an embodiment of the present invention;

[0058] Figure 4 This is one of the schematic diagrams of the target stove provided in the embodiments of the present invention;

[0059] Figure 5 This is the second schematic diagram of the target stove provided in the embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of the camera calibration device for a cooking apparatus provided in an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0062] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0063] In the description of the embodiments of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0064] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0065] In embodiments of the present invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0066] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0067] The following is combined Figures 1 to 5The present invention describes a camera calibration method for a cooking device according to an embodiment of the invention. The execution subject of the method can be a controller on the device side, or a cloud or an edge server.

[0068] like Figure 1 As shown, the camera calibration method of the cooking device of the present invention includes steps 110 to 140.

[0069] Step 110: Input the target image of the target stove into the target recognition model to obtain the first coordinates of multiple target objects in the target stove.

[0070] By capturing a target image of the target stovetop using a camera installed on the cooking equipment, the cookware, stovetop, ingredients, and cooking behavior in the target image can be detected and identified. The identified information can then be used to assist the user in cooking.

[0071] The cooking equipment can be a range hood used in conjunction with the target stove, or the cooking equipment can be the target stove itself. The camera is installed in a position that can capture all target objects on the target stove.

[0072] The target objects are fixed items such as stoves, switches, or brand logos on the target stove.

[0073] A target recognition model is constructed by using a target detection algorithm based on a deep learning framework. This model can identify and output the target object and its coordinates in a target image.

[0074] Take a range hood as an example of a cooking appliance.

[0075] like Figure 2 As shown, the camera 10 is installed on the range hood 30. The camera 10 can capture the target image of the target stove 20. The target image is input into the target recognition model, and the first coordinates of the target objects such as the first stove 21, the second stove 22, the first switch 23 and the second switch 24 on the target stove 20 can be input.

[0076] In practice, the target recognition model needs to be trained first, and then the target image captured by the camera to be calibrated is input into the target recognition model for recognition to obtain the coordinates of the target object.

[0077] The target recognition model is trained using sample images as samples and pre-determined coordinates of the sample objects corresponding to the sample images as sample labels.

[0078] The target detection algorithm used to build the target recognition model can be selected based on the application scenario and the requirements of the computing hardware. The target detection algorithm can be an anchor-based algorithm, such as YOLO, SSD, RetinaNet and Faster R-CNN, which are various single-stage or two-stage detection algorithms; or it can be an anchor-free algorithm, such as CornerNet, CenterNet and FCOS.

[0079] When deploying target recognition models in embedded chips, lightweight recognition and detection models such as MobileNet, ShuffleNet, and GhostNet can be selected.

[0080] The target recognition model outputs the first coordinates of the target object, which are the initial coordinates of the target object and include some coordinates that do not belong to the target object.

[0081] It is understandable that by taking multiple images of the target stove with a camera, multiple image information of the target stove can be obtained. The more image information there is, the higher the accuracy of the first coordinate of the target object output by the target recognition model.

[0082] In practice, each image of a target stove can be input into the trained target recognition model, which will output the first coordinates of multiple target objects respectively; alternatively, multiple images of target stoves can be input into the trained target recognition model in batches, which will output the first coordinates of multiple target objects respectively.

[0083] Step 120: Perform cluster analysis on the first coordinates to obtain the second coordinates of multiple target objects.

[0084] In this step, the first coordinate and the second coordinate are the sets of coordinates of each point in multiple target objects in the target image. Cluster analysis is performed on the first coordinates of multiple target objects. The first coordinates are processed using a clustering algorithm, and the coordinates of points in the first coordinates that do not belong to the corresponding target objects are discarded to obtain the second coordinates of multiple target objects.

[0085] The second coordinate is obtained by iteratively processing the first coordinate, discarding coordinates that do not belong to the corresponding target object, thus obtaining the accurate coordinates of multiple target objects.

[0086] In practice, clustering algorithms such as K-MEANS, K-MEDOIDS, and CLARANS can be used to perform cluster analysis on the first coordinate.

[0087] Understandably, cluster analysis of the first coordinates of multiple target stove images yields more accurate second coordinates with a larger number of target stove images.

[0088] Step 130: Obtain the spatial location features of the target stove.

[0089] Spatial location features refer to the coordinates of a point on a real object in the world coordinate system. Spatial location features of the target stove refer to the spatial location features of multiple target objects on the target stove.

[0090] In this step, the spatial position characteristics of each target object can be obtained based on the size information and relative positional relationship of each target object on the target stove.

[0091] In practice, the size information and relative position of each target object can be obtained by reading the model or hardware design parameters of the target stove; alternatively, the size information and relative position of the target objects on the target stove corresponding to the model can be downloaded from the cloud by the user inputting the model of the target stove; or the size information and relative position of the target objects on the target stove can be directly input based on actual measurements.

[0092] It should be noted that due to the different layout designs of cooking equipment, there may be deviations in the mounting height between the camera and the target stove. Multiple verification methods can be used to statistically correct the spatial position characteristics of each target object on the target stove.

[0093] Step 140: Based on the second coordinates and spatial location features, obtain the camera calibration parameters of the cooking device's camera.

[0094] In image measurement and machine vision applications, to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, it is necessary to establish a geometric model of camera imaging. These geometric model parameters are the camera calibration parameters.

[0095] like Figure 3 As shown, the camera calibration parameters are calculated by calculating the correspondence between multiple corresponding world 3D coordinate points P and corresponding 2D coordinate points p on the image.

[0096] The second coordinate is the coordinate of the two-dimensional coordinate point of the target object in the target image, and the spatial position feature is the coordinate of the three-dimensional coordinate point of the target object on the target stove. The camera calibration parameters of the camera can be obtained by calculating based on the second coordinate and spatial position feature corresponding to the target object.

[0097] It is understandable that the camera is fixedly installed on the cooking equipment. The camera calibration parameters obtained by using the second coordinate and spatial position features are the camera calibration parameters of the camera at the current position. After the camera position changes, the camera calibration parameters need to be recalculated.

[0098] In this step, by using the second coordinates and spatial position features of the target object, camera calibration parameters can be calculated, and the camera can be calibrated to correct lens distortion, generate a corrected image, and reconstruct a more accurate 3D scene.

[0099] Understandably, by using the second coordinates and spatial location features of the target object to calculate the camera calibration parameters and calibrate the camera, the image information acquired by the camera becomes more accurate. This allows for more accurate detection and identification of information such as pots, stoves, ingredients, and cooking activities on the target stove, thus assisting in cooking and enhancing its enjoyment.

[0100] According to the camera calibration method for cooking equipment provided by the present invention, target objects on the target stove are identified through deep learning, cluster analysis is used to improve the accuracy of the target object's coordinates, and the camera calibration parameters of the camera are determined based on the second coordinates and spatial location characteristics of the target object. This helps to obtain more accurate images of the target stove, improve the accuracy of subsequent visual tasks, accurately identify information such as pots, stoves, ingredients and cooking behavior on the target stove, assist in cooking, and enhance the fun of cooking.

[0101] In some embodiments, step 140 includes: constructing an equation for camera calibration parameters based on the second coordinates and spatial location features of the target object, performing a matrix transformation on the equation, and calculating the camera calibration parameters using the least squares method.

[0102] In actual execution, multiple feature points in one or more target objects are selected, and multiple sets of equations containing camera calibration parameters are constructed based on the second coordinates of these feature points on the target image and their spatial position features on the target stove.

[0103] Feature points can be the center point or calibration point of the target object, such as the center point of the stove, the calibration points at the four corners of the stove frame, the center point of the stove switch, or the calibration point set on the target stove.

[0104] After constructing multiple sets of equations, matrix transformations are performed on these equations, and the least squares method is used to solve them, thereby obtaining the camera calibration parameters corresponding to the camera.

[0105] In this embodiment, when using the least squares method to solve the problem, selecting more feature points and finding the best function match for the data by minimizing the sum of squared errors can yield more accurate camera calibration parameters.

[0106] In some embodiments, the camera calibration parameters of the camera include intrinsic and extrinsic parameters.

[0107] Among them, the intrinsic parameters are only related to the properties of the camera itself, and are the parameters used by the camera to transform from three-dimensional camera coordinates to two-dimensional image coordinates.

[0108] External parameters are related to the camera's installation location and are used when the camera transforms from three-dimensional world coordinates to two-dimensional camera coordinates.

[0109] Optical distortion is the error in the position of a point that deviates from the ideal position due to the design, manufacturing, and assembly of the camera objective system. Distortion parameters are used to correct the optical distortion of the camera and are considered intrinsic parameters of the camera.

[0110] The equation constructed based on the second coordinates of the feature points on the target object and their corresponding spatial location features is as follows:

[0111]

[0112] in, The second coordinate of the feature point on the target object. The spatial location features corresponding to the feature points. [R|t] is the intrinsic parameter, [R|t] is the extrinsic parameter, R and t are homogeneous transformation matrices, which are the transformation matrices when the camera transforms from three-dimensional world coordinates to two-dimensional camera coordinates, x0 and y0 are the origin of the second coordinates, that is, at the origin of the two-dimensional camera coordinates, and α and s are the distortion parameters of the intrinsic parameter.

[0113] The equation obtained after matrix transformation is:

[0114]

[0115] The second coordinate of the feature point on the target object. The spatial location features corresponding to the feature points. To calibrate the camera parameters.

[0116] In practice, six feature points are selected. Based on the six sets of second coordinates of these feature points and their corresponding spatial location features, the following can be achieved: The 11 m-parameters in the image are solved using the least squares method, thereby obtaining the intrinsic, extrinsic, and distortion parameters of the camera.

[0117] In some embodiments, step 120, performing cluster analysis, may include: obtaining the number of target objects on the target stove, using this number as the number of clusters for cluster analysis, and using this number of clusters to perform cluster analysis on the first coordinates to obtain the second coordinates of multiple target objects.

[0118] The number of multiple target objects on the target stove is used as the number of clusters for cluster analysis. The first coordinates of multiple target objects are divided into different clusters, processed, and the corresponding second coordinates are obtained by clustering.

[0119] Using the number of target objects on the target stove as the cluster number, cluster analysis can effectively distinguish the first coordinates of different target objects and improve the accuracy of the obtained second coordinates.

[0120] like Figure 4 As shown, there are two stoves on the target stove. Taking the stoves as the target objects, when analyzing the first coordinates of the stoves, the number of stoves is used as the cluster number, and the first coordinates are divided into the first coordinates of different stoves.

[0121] like Figure 5 As shown, after performing cluster analysis on the first coordinates of different stoves, the second coordinates of different stoves are obtained, such as the second coordinates of target objects like the first stove 21, the second stove 22, the first switch 23, and the second switch 24.

[0122] In some embodiments, the number of target objects on the target stove can be obtained by means of the device information of the target stove or the output of the target recognition model.

[0123] First, based on the equipment information of the target stove, the number of target objects is obtained.

[0124] In practice, the device information of the target stove can be read directly, or the user can input or read the model number of the target stove and download the corresponding device information from the cloud. Alternatively, the device information can be directly input by the user.

[0125] The equipment information may include the type and number of target objects such as cooktops, switches, or brand logos on the target cooktop.

[0126] Secondly, the target image of the target stove is input into the target recognition model, and the target recognition model outputs the number of target objects.

[0127] A target recognition model is constructed by using a target detection algorithm based on a deep learning framework. This model can identify target objects in a target image and output the type and number of target objects.

[0128] In practice, the target recognition model can be trained using sample images as samples and pre-determined types of sample objects corresponding to the sample images as sample labels.

[0129] Understandably, by capturing multiple images of the target stove with a camera, multiple image information of the target stove can be obtained. The more image information there is, the higher the accuracy of the target recognition model in outputting the type and number of target objects.

[0130] In practice, the number of target objects output by each target stove image can be counted in the trained target recognition model, and the number of objects that appear most frequently can be selected as the number of clusters for cluster analysis.

[0131] In some embodiments, after determining the number of clusters based on the number of target objects, cluster analysis is performed on the first coordinates of the target objects based on the number of clusters to obtain the second coordinates.

[0132] In this embodiment, the first coordinates of multiple target objects are divided according to the number of clusters determined by the number of clusters, and the first coordinates of the target objects are divided into different first coordinates of the target objects.

[0133] For a given target object, obtain the first coordinates of the two diagonal points in the first coordinate system of the target object, calculate the average of the first coordinates of the two diagonal points, and obtain the coordinates of the center point of the target object.

[0134] For multiple target objects on the target stove, the center point coordinates of different target objects are calculated based on the first coordinates of the two diagonal points of different target objects.

[0135] In actual implementation, multiple target stove images are captured by cameras, which can obtain the coordinates of multiple center points of the same target object. Cluster analysis is then performed on the coordinates of the multiple center points of the target object to obtain the coordinates of the target center point.

[0136] The distance between the coordinates of each point in the first coordinate system and the coordinates of the target center point is calculated. Using this distance as a criterion, the points in the first coordinate system are clustered, and valid points are selected and retained while invalid points are discarded.

[0137] Based on the principle that objects within the same cluster have high similarity, while objects in different clusters have low similarity, the distance value is compared with a preset distance threshold. If the distance value calculated for a point in the first coordinate is greater than the distance threshold, it indicates that the point has low similarity with objects in its cluster, and the coordinate corresponding to that point is discarded.

[0138] If the distance value calculated for a point in the first coordinate system is not greater than the distance threshold, it indicates that the point has a high similarity to objects within the same cluster, and the coordinates corresponding to that point are retained.

[0139] The distance between the coordinates of all points in the first coordinate system and the coordinates of the target center point is calculated and compared with a preset distance threshold. The retained first coordinate system is the second coordinate system of the target object.

[0140] The following is a specific example.

[0141] The target objects on the target stove are two stoves. Sixteen target images of the target stove are obtained through a camera. These 16 target images are then input into the target recognition model to obtain the first coordinates of the two stoves.

[0142] For a single stove, the first coordinates of 16 points with two opposite corners were obtained statistically. in, and The top-left x-coordinate, top-left y-coordinate, bottom-right x-coordinate, and bottom-right y-coordinate represent the position coordinates of the i-th stove, where i = 1...n. This represents the n stoves detected in all 16 images, where n is 2.

[0143] Among them, the top left corner and the bottom right corner are the diagonal points of one stove.

[0144] For each stove location Calculate the coordinates of the center point of each stove.

[0145] For 16 target images, the coordinates of 16 center points for each stove are calculated, resulting in a total of 32 center point coordinates for the two stoves. Cluster analysis is then performed on these 32 center point coordinates to obtain the target center point coordinates for each stove after clustering, resulting in a total of two target center point coordinates.

[0146] Based on the coordinates of the target center point, the first coordinates of the stove in each target image are grouped. A preset distance threshold is set. If the distance between the first coordinate of a point of the stove in the group and the coordinates of the target center point is greater than the distance threshold, the first coordinate of that point is discarded. If the distance is not greater than the distance threshold, the first coordinate of that point is retained.

[0147] The retained first coordinates are used as the second coordinates of the stove. The average value of the center coordinates of the stove is recalculated based on the second coordinates, and then the size parameters of the stove, such as the average width and height of the stove frame, are calculated. Combined with the spatial position characteristics of the stove, the camera is calibrated.

[0148] The present invention also provides a camera calibration system for a cooking device, comprising: a cooking device, a camera, a target stove, and a controller.

[0149] The camera is installed on the cooking equipment to capture a target image of the target stove.

[0150] The cooking equipment can be a range hood used in conjunction with the target stove, or it can be the target stove itself. The target stove is located within the camera's field of view, and the camera is installed in a position that can capture the entire area of ​​the target stove.

[0151] The target stove has multiple target objects, which can be fixed objects such as stoves, switches, or brand logos.

[0152] The controller is electrically connected to the camera. The camera can be calibrated using the aforementioned camera calibration method for cooking equipment. After calibration, the image information acquired by the camera is more accurate, enabling more precise detection and identification of information such as pots, stoves, ingredients, and cooking behaviors on the target stove. The identified information can then be used to assist users in cooking and enhance the enjoyment of cooking.

[0153] The camera calibration system for cooking equipment provided by this invention identifies target objects on the target stove through deep learning, improves the accuracy of the target object's coordinates by using cluster analysis, and determines the camera calibration parameters of the camera based on the target object's second coordinates and spatial location characteristics. This helps to obtain more accurate images of the target stove, improves the accuracy of subsequent visual tasks, accurately identifies information such as pots, stoves, ingredients, and cooking behaviors on the target stove, assists in cooking, and enhances the enjoyment of cooking.

[0154] The camera calibration device for cooking equipment provided in the embodiments of the present invention will be described below. The camera calibration device for cooking equipment described below can be referred to in correspondence with the camera calibration method for cooking equipment described above.

[0155] like Figure 6 As shown, the camera calibration device of the cooking apparatus of the present invention includes:

[0156] The first processing module 610 is used to input the target image of the target stove into the target recognition model and obtain the first coordinates of multiple target objects in the target stove output by the target recognition model.

[0157] The second processing module 620 is used to perform cluster analysis on the first coordinates to obtain the second coordinates of multiple target objects;

[0158] The acquisition module 630 is used to acquire the spatial location features of multiple target objects on the target stove.

[0159] The calculation module 640 is used to obtain the camera calibration parameters of the camera of the cooking device based on the second coordinates and spatial position features;

[0160] The target image is acquired through a camera, and the target recognition model is trained using sample images as samples and pre-determined coordinates of the sample objects corresponding to the sample images as sample labels.

[0161] The camera calibration device for cooking equipment provided by the present invention identifies target objects on the target stove through deep learning, improves the accuracy of the target object's coordinates by using cluster analysis, and determines the camera calibration parameters of the camera based on the second coordinates and spatial location characteristics of the target object. This helps to obtain more accurate images of the target stove, improves the accuracy of subsequent visual tasks, accurately identifies information such as pots, stoves, ingredients, and cooking behaviors on the target stove, assists in cooking, and enhances the fun of cooking.

[0162] In some embodiments, the calculation module 640 is used to obtain multiple sets of equations about camera calibration parameters based on the second coordinates and spatial position features; perform matrix transformations on the multiple sets of equations; and obtain the camera calibration parameters by the least squares method.

[0163] In some embodiments, camera calibration parameters include intrinsic and extrinsic parameters, and the equation is as follows:

[0164]

[0165] in, The second coordinate, For spatial location features, α is the intrinsic parameter, [R|t] is the extrinsic parameter, R and t are homogeneous transformation matrices, x0 and y0 are the origin of the second coordinates, and α and s are the distortion parameters of the intrinsic parameter.

[0166] The equation obtained after matrix transformation is:

[0167]

[0168] The second coordinate, For spatial location features, To calibrate the camera parameters.

[0169] In some embodiments, the second processing module 620 is used to obtain the number of multiple target objects, use the number of multiple target objects as the number of clusters for cluster analysis, and perform cluster analysis on the first coordinates according to the number of clusters to obtain the second coordinates.

[0170] In some embodiments, the second processing module 620 obtains the number of multiple target objects, including: obtaining device information of the target stove; and obtaining the number of multiple target objects based on the device information.

[0171] In some embodiments, the second processing module 620 obtains the number of multiple target objects, including:

[0172] The target image of the target stove is input into the target recognition model to obtain the number of multiple target objects in the target stove output by the target recognition model.

[0173] In some embodiments, the second processing module 620 performs cluster analysis on the first coordinates based on the number of clusters to obtain the second coordinates, including:

[0174] Based on the number of clusters, the first coordinates are divided into the first coordinates of different target objects; the first coordinates of the two diagonal points of the first coordinates of different target objects are obtained, and the center point coordinates of the target objects are calculated; cluster analysis is performed on the center point coordinates to obtain the target center point coordinates; the distance value between each point in the first coordinates of the target objects and the target center point coordinates is calculated; the distance value is compared with a distance threshold, and points with a distance value greater than the distance threshold are discarded; points with a distance value not greater than the distance threshold are retained to obtain the second coordinates.

[0175] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communication interface 720, a memory 730, and a communication bus 740. The processor 710, communication interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a camera calibration method for the cooking device. This method includes: inputting a target image of the target stove into a target recognition model to obtain the first coordinates of multiple target objects in the target stove output by the target recognition model; performing cluster analysis on the first coordinates to obtain the second coordinates of the multiple target objects; acquiring the spatial location features of the multiple target objects in the target stove; and obtaining the camera calibration parameters of the cooking device's camera based on the second coordinates and spatial location features. The target image is acquired by a camera, and the target recognition model is trained using sample images as samples and pre-determined coordinates of sample objects corresponding to the sample images as sample labels.

[0176] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] Furthermore, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the camera calibration method for cooking equipment provided in the above-described method embodiments. The method includes: inputting a target image of a target stove into a target recognition model to obtain the first coordinates of multiple target objects in the target stove output by the target recognition model; performing cluster analysis on the first coordinates to obtain the second coordinates of the multiple target objects; acquiring the spatial location features of the multiple target objects in the target stove; and obtaining the camera calibration parameters of the camera of the cooking equipment based on the second coordinates and the spatial location features. The target image is acquired by a camera, and the target recognition model is trained using sample images as samples and pre-determined coordinates of sample objects corresponding to the sample images as sample labels.

[0178] On the other hand, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the camera calibration method for the cooking equipment provided in the above embodiments. The method includes: inputting a target image of a target stove into a target recognition model to obtain the first coordinates of multiple target objects in the target stove output by the target recognition model; performing cluster analysis on the first coordinates to obtain the second coordinates of the multiple target objects; acquiring the spatial location features of the multiple target objects in the target stove; and obtaining camera calibration parameters of the camera of the cooking equipment based on the second coordinates and the spatial location features. The target image is acquired by a camera, and the target recognition model is trained using sample images as samples and pre-determined coordinates of sample objects corresponding to the sample images as sample labels.

[0179] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for calibrating a camera in a cooking device, characterized in that, include: The target image of the target stove is input into the target recognition model to obtain the first coordinates of multiple target objects in the target stove output by the target recognition model; wherein, the target object is an object or item with a fixed position on the target stove. Perform cluster analysis on the first coordinates to obtain the second coordinates of the multiple target objects; Obtain the spatial position features of the multiple target objects in the target stove; Based on the second coordinates and the spatial location features, the camera calibration parameters of the cooking device's camera are obtained; The target image is acquired by the camera, and the target recognition model is trained using sample images as samples and pre-determined coordinates of sample objects corresponding to the sample images as sample labels. The step of performing cluster analysis on the first coordinates to obtain the second coordinates of the multiple target objects includes: The number of the multiple target objects is obtained, and the number of the multiple target objects is used as the number of clusters for cluster analysis; Based on the number of clusters, the first coordinates are subjected to cluster analysis to obtain the second coordinates; The step of performing cluster analysis on the first coordinates based on the number of clusters to obtain the second coordinates includes: Based on the number of clusters, the first coordinates are divided into the first coordinates of different target objects; By obtaining the first coordinates of two diagonal points in the first coordinates of different target objects, the coordinates of the center point of the target object are obtained; Cluster analysis is performed on the coordinates of the center point to obtain the coordinates of the target center point; Calculate the distance between each point in the first coordinate system of the target object and the coordinate system of the center point of the target object; The distance value is compared with a distance threshold, and points whose distance value is greater than the distance threshold are discarded. The points whose distance values ​​are not greater than the distance threshold are retained to obtain the second coordinates.

2. The camera calibration method for cooking equipment according to claim 1, characterized in that, The process of obtaining camera calibration parameters for the cooking device's camera based on the second coordinates and the spatial location features includes: Based on the second coordinates and the spatial location features, multiple sets of equations are obtained regarding the camera calibration parameters; The camera calibration parameters are obtained by performing matrix transformations on multiple sets of equations and using the least squares method.

3. The camera calibration method for cooking equipment according to claim 2, characterized in that, The camera calibration parameters include intrinsic and extrinsic parameters, and the equation is: ； in, The second coordinate is... For the aforementioned spatial location features, The internal reference, The extrinsic parameter, and It is a homogeneous transformation matrix. and The origin of the second coordinate system is [the point where the coordinates are located]. and The distortion parameter of the intrinsic parameter; The equation obtained after matrix transformation is: ； The second coordinate is... For the aforementioned spatial location features, The camera is calibrated.

4. The camera calibration method for cooking equipment according to claim 1, characterized in that, Obtaining the number of the plurality of target objects includes: Obtain the device information of the target stove; Based on the device information, the number of the plurality of target objects is obtained.

5. The camera calibration method for cooking equipment according to claim 1, characterized in that, Obtaining the number of the plurality of target objects includes: The target image of the target stove is input into the target recognition model to obtain the number of the multiple target objects in the target stove as output by the target recognition model.

6. A camera calibration system for a cooking device, characterized in that, include: Cooking equipment; A camera is mounted on the cooking device; The target stove is equipped with multiple target objects and is located within the shooting area of ​​the camera. A controller electrically connected to the camera, the controller being configured to control the camera to perform calibration based on the camera calibration method of the cooking device according to any one of claims 1-5.

7. A camera calibration device for a cooking apparatus, characterized in that, include: The first processing module is used to input the target image of the target stove into the target recognition model to obtain the first coordinates of multiple target objects in the target stove output by the target recognition model; wherein, the target object is an object or item with a fixed position on the target stove. The second processing module is used to perform cluster analysis on the first coordinates to obtain the second coordinates of the multiple target objects; The acquisition module is used to acquire the spatial position features of the multiple target objects in the target stove; The calculation module is used to obtain the camera calibration parameters of the camera of the cooking device based on the second coordinates and the spatial position features; The second processing module is used to obtain the number of multiple target objects, and use the number of multiple target objects as the cluster number for cluster analysis; based on the cluster number, the first coordinate is subjected to cluster analysis to obtain the second coordinate; The second processing module performs cluster analysis on the first coordinates based on the number of clusters to obtain the second coordinates. This includes: dividing the first coordinates into the first coordinates of different target objects based on the number of clusters; obtaining the first coordinates of the two diagonal points of the first coordinates of different target objects, and calculating the center point coordinates of the target objects; performing cluster analysis on the center point coordinates to obtain the target center point coordinates; calculating the distance between each point in the first coordinates of the target objects and the target center point coordinates; comparing the distance values ​​with a distance threshold, discarding points whose distance values ​​are greater than the distance threshold; and retaining points whose distance values ​​are not greater than the distance threshold to obtain the second coordinates. The target image is acquired by the camera, and the target recognition model is trained using sample images as samples and pre-determined coordinates of sample objects corresponding to the sample images as sample labels.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the camera calibration method for the cooking device as described in any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the camera calibration method for the cooking apparatus as described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the camera calibration method for the cooking device as described in any one of claims 1 to 5.

Images