Oven and image processing method

Abstract

The invention provides an oven and an image processing method, and relates to the technical field of intelligent household appliances. In order to improve the transmission speed and the restoration speed of food material photos by the oven, the oven provided by the invention comprises a shooting device and a processor, and the oven is connected with a cloud end. The camera device is configured to shoot an internal structure of a cavity of the oven to obtain a first image. The first image comprises a first area, the first area comprises a second area, and the second area is an area which is covered with food materials and can be recognized. The processor is configured to receive the first image and extract image information of a plurality of pixel points in the second area and coordinates of a plurality of vertexes; and the image information and the coordinates are sent to the cloud. The cloud is configured to receive the coordinates of the plurality of vertexes of the second area and the image information, and obtain a third area in the restored image according to the coordinates of the plurality of vertexes of the second area. And correspondingly filling the image information of the plurality of pixel points into the plurality of pixel points in the third region one by one to obtain a restored image.

Description

Technical Field

[0001] The present invention relates to the technical field of smart home appliances, and in particular to an oven and an image processing method. Background Art

[0002] For ovens equipped with cameras, image recognition is a relatively important link. Specifically, the camera takes a picture of the inside of the oven, and the oven sends the image information of the picture in the form of a data stream to the cloud connected to the oven. The cloud restores the data to obtain the picture taken by the camera, and then recognizes the picture. The picture is composed of multiple pixels. In the process from the camera taking the picture to the cloud connected to the oven, a large amount of useless pixel data that is not related to the ingredients is also transmitted, such as the pixel data of the baking tray and the pixel data of some structures inside the oven. Compared with the pixels occupied by the ingredients, these useless pixels occupy a larger proportion of the entire photo. Since the photos taken by the camera are in the form of a data stream, that is, the image information of the pixels on the picture is transmitted to the cloud, it is necessary to extract the image information of the pixels on the picture, and then restore the picture according to the image information of the pixels. The existence of a large number of useless pixels will lead to a longer time for the pixel extraction process and a longer time for the cloud to restore the picture. Summary of the invention

[0003] The embodiments of the present invention provide an oven and an image processing method, wherein the oven can reduce the transmission of useless signals and improve the accuracy of the oven in identifying food.

[0004] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0005] In a first aspect, the oven provided by the present invention includes: a cavity, a shooting device, a processor, and a cloud connected to the oven. The camera device is configured to shoot the internal structure of the cavity of the oven to obtain a first image. The first image includes a first area, the first area includes a second area, and the second area is an area covered with food and can be identified. The processor is configured to receive the first image, extract the image information of multiple pixels in the second area in the first image and obtain the coordinates of multiple vertices of the second area; and send the image information of the multiple pixels and the coordinates of the multiple vertices of the second area to the cloud. The cloud is configured to receive the coordinates of the multiple vertices of the second area and the image information of the multiple pixels, and obtain the third area in the initial restored image according to the coordinates of the multiple vertices of the second area. The image information of the multiple pixels is filled into the multiple pixels in the third area one by one to obtain a restored image.

[0006] After the camera captures the first image, the oven does not transmit the image information of all pixels of the first image, but selects the second area in the first image, that is, the area covered with food and can be identified, and transmits the image information of the pixels in the second area and the vertex coordinates of the second area. After receiving the image information of the pixels and the vertex coordinates of the second area, the cloud restores the image. During the whole process, the data to be transmitted is more streamlined and more efficient than transmitting all pixel information to the cloud, and the bandwidth and resources occupied are less and the speed is faster, so that more resources can be allocated to the identification of food and the judgment of the maturity of food. Therefore, the accuracy of intelligent identification in the cloud is also greatly improved.

[0007] As a possible implementation, the processor is configured to extract image information of multiple pixels located in the second area in the first image. Specifically, the processor is configured to connect adjacent coordinates in sequence with straight lines according to the coordinates of the vertices of the second area pre-calibrated in the first area, obtain the range of the second area, and extract image information of multiple pixels located in the second area. The coordinates of the vertices of the pre-calibrated second area in the first area are calibrated in the first coordinate system, and the origin of the first coordinate system is a vertex of the first area.

[0008] As a possible implementation, the processor is configured to obtain the coordinates of multiple vertices of the second area. Specifically configured as: establish a second coordinate system in the first image. The second area is a trapezoid, and the multiple vertices of the second area include a first vertex, a second vertex, a third vertex and a fourth vertex. The first vertex and the second vertex are two endpoints of the upper base of the trapezoid, the third vertex and the fourth vertex are two endpoints of the lower base of the trapezoid, and the second vertex and the third vertex are adjacent. The origin of the coordinate system is the intersection of the perpendicular line of the lower base of the trapezoid and the extension line of the upper base of the trapezoid, and the perpendicular line of the lower base of the trapezoid passes through the fourth vertex. The horizontal axis of the coordinate system is the straight line where the upper base is located, and the vertical axis of the coordinate system is the perpendicular line of the lower base of the trapezoid. The unit length of the coordinate axis is the size of the pixel point. The coordinates of the first vertex (x1, y1), the coordinates of the second vertex (x2, y1), the coordinates of the third vertex (x3, y2) and the coordinates of the fourth vertex (x4, y2) are obtained in the second coordinate system.

[0009] As a possible implementation method, the cloud is configured to obtain the third area in the initial restored image based on the coordinates of multiple vertices of the second area, and is specifically configured to: establish a third coordinate system in the initial restored image. In the third coordinate system, the positions of the four vertices of the third area are determined based on the coordinates of the first vertex (x1, y1), the coordinates of the second vertex (x2, y1), the coordinates of the third vertex (x3, y2), and the coordinates of the fourth vertex (x4, y2) in the second coordinate system. Based on the four vertices of the third area, the range of the third area is obtained.

[0010] As a possible implementation method, the cloud is configured to fill the image information of multiple pixels one by one into the third area to obtain a restored image, including being specifically configured to: obtain the number of rows k in the third area; obtain the coordinates of the starting pixel points of each row in the third area; obtain the number of pixels contained in each row in the third area; and fill the image information of multiple pixels into the multiple pixels of the third area in sequence according to the coordinates of the starting pixel points of each row and the number of pixels contained in each row.

[0011] As a possible implementation manner, the cloud is configured to obtain the number of rows k of the third area, and is specifically configured to: subtract the ordinate of the fourth vertex from the ordinate of the first vertex to obtain the number of rows k.

[0012] As a possible implementation method, the cloud is configured to obtain the coordinates of the starting pixel point of each row of the third area, specifically configured as: the coordinates of the starting pixel point of the first row are the coordinates of the first vertex (x1, y1); the coordinates of the starting point of the nth row are (x1-[x1*(n-1)) / y2], y1+n-1), where n is 2, 3, 4...k-1, k.

[0013] As a possible implementation method, the cloud is configured to obtain the number of pixels contained in each row of the third area, specifically configured as follows: the number of pixels in the first row is x2-x1; the number of pixels in the nth row is 2x1(n-1) / y2.

[0014] In a second aspect, the present invention also provides an image processing method, which includes: photographing the internal structure of the cavity of an oven to obtain a first image. The first image includes a first area, the first area includes a second area, and the second area is an area covered with food and can be identified. Receive the first image, in the first image, extract the image information of multiple pixels located in the second area and obtain the coordinates of multiple vertices of the second area. And send the image information of the multiple pixels and the coordinates of the multiple vertices of the second area to the cloud. Receive the coordinates of the multiple vertices of the second area and the image information of the multiple pixels. According to the coordinates of the multiple vertices of the second area, obtain the third area in the initial restored image. Fill the image information of the multiple pixels into the multiple pixels in the third area one by one to obtain a restored image.

[0015] For the technical effects of the second aspect, reference may be made to the relevant description of the first aspect mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0017] Figure 1 A schematic diagram of the composition of an oven provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of a first area and a second area provided in an embodiment of the present application;

[0019] Figure 3 A physical schematic diagram of an oven provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of a camera installation position provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of another camera installation position provided in an embodiment of the present application;

[0022] Figure 6 A schematic diagram of a camera shooting area provided in an embodiment of the present application;

[0023] Figure 7 A schematic diagram of the position relationship of a second area in a first image provided in an embodiment of the present application;

[0024] Figure 8 A schematic diagram of vertex coordinates of a third region provided in an embodiment of the present application;

[0025] Fig. 9 A schematic diagram of a third region provided in an embodiment of the present application;

[0026] Fig.10 A schematic diagram of image restoration comparison provided in an embodiment of the present application;

[0027] Fig.11 A cloud configuration flow chart provided for an embodiment of the present application;

[0028] Fig.12 A schematic diagram of a pixel arrangement constituting an image displayed in a third area provided in an embodiment of the present application;

[0029] Fig.13 A cloud-specific configuration flow chart provided in an embodiment of the present application;

[0030] Fig.14 Another cloud specific configuration flow chart provided for an embodiment of the present application;

[0031] Fig.15 A schematic diagram of calculating the coordinates of a starting pixel point provided in an embodiment of the present application;

[0032] Fig.16 Another schematic diagram of calculating the coordinates of the starting pixel provided in the embodiment of the present application;

[0033] Fig.17 Another cloud specific configuration flow chart provided for an embodiment of the present application;

[0034] Fig.18 A schematic diagram of selecting a second region provided in an embodiment of the present application;

[0035] Fig.19 Another schematic diagram of second area selection provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0038] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0039] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, when describing the pipeline, the "connected" and "connection" used in the present invention have the meaning of conduction. The specific meaning needs to be understood in conjunction with the context.

[0040] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0041] With the continuous advancement of technology, smart home appliances are constantly being updated and iterated. As a cooking tool that people often use in their daily lives, ovens have derived a series of other functions in addition to the original baking function. For example, a camera is provided in the oven, and the camera can capture images of the inside of the oven, so that the changes of food during the entire baking process can be obtained based on the images.

[0042] For ovens equipped with a camera, image recognition is a relatively important link. Specifically, the camera takes a picture of the inside of the oven, and the oven sends the image information of the picture in the form of a data stream to the cloud connected to the oven. The cloud restores the data to obtain the picture taken by the camera, and then recognizes the picture. The recognition mainly includes the judgment of the ingredients and the judgment of the maturity of the ingredients.

[0043] The cloud's judgment of the ingredients and the maturity of the ingredients are based on the state, color and other factors of the ingredients in the picture. The picture is composed of multiple pixels. In the process from the shooting device to the picture to the transmission to the cloud connected to the oven, a large amount of useless pixel data unrelated to the ingredients is also transmitted, such as the pixel data of the baking tray and the pixel data of some structures inside the oven. Compared with the pixels occupied by the ingredients, these useless pixels occupy a larger proportion of the entire photo. Since the photos taken by the shooting device are in the form of data streams, that is, the image information of the pixels on the picture is transmitted to the cloud, it is necessary to extract the image information of the pixels on the picture, and then restore the picture based on the image information of the pixels. The existence of a large number of useless pixels will lead to a longer time for the pixel extraction process and a longer time for the cloud to restore the picture.

[0044] like Figure 1 As shown, attached Figure 1 The composition of an oven provided by the present application is shown in FIG. Figure 1 The oven 100 includes a cavity 1, a camera 2, a processor 3, and a cloud 4, and the cloud 4 is connected to the oven 100. A baking tray is arranged in the cavity 1 of the oven 100, and the camera 2 is configured to photograph the internal structure of the cavity 100 of the oven 100 to obtain a first image. The first image shows the internal structure of the oven 100, the baking tray, and the food placed on the baking tray. The area where the entire picture displayed on the first image is located is called the first area.

[0045] Since the cloud 4 identifies the food and determines the maturity of the food based on the state of the food, such as color, shape, etc., there are many useless images in the first image except the food itself, so a second area is divided from the first area displayed in the first image, and the second area is an area covered with food and can be identified. Being able to be identified means that the content displayed in the area is the bottom line for the cloud to identify the food and judge the maturity of the food. If the area is further reduced, the cloud will not be able to correctly identify the food or accurately judge the maturity of the food.

[0046] like Figure 2 As shown, Figure 2 The difference between the first area and the second area is shown. Figure 2 , attached Figure 2 The rectangle shown in the figure represents the first image captured by the camera 2. The first image includes the first area 21, and the first area 21 includes the second area 211. All the contents displayed on the first image are indicated by vertical lines filling the inside of the rectangle, where the vertical lines cover the entire rectangular area, which means that the pictures displayed on the first image all belong to the first area 21. The second area 211 is shown in the figure below. Figure 2The middle is a trapezoidal area, and the content displayed in the second area 211 is indicated by filled with oblique lines. This area is an area covered with food and can be identified.

[0047] Combined with Figure 3 , Figure 3 A physical picture of an oven provided in an embodiment of the present application is shown. The oven 100 includes a cavity 1, a camera 2, a processor 3, and a cloud 4. The camera 2 captures a first image and transmits it to the processor 3. The processor 3 inside the oven 100 is configured to receive the first image, extract image information of multiple pixels located in the second area 211 in the first image, and send the image information of the multiple pixels to the cloud 4.

[0048] An image is composed of multiple pixels, and the image information of the pixels includes the RGB grayscale values ​​of the pixels. The corresponding image can be restored through the image information of the multiple pixels. The multiple pixels on the second area 211 constitute the display screen in the second area 211. By extracting the image information of the multiple pixels, that is, extracting the RGB grayscale values ​​of the multiple pixels, the corresponding pixels can be restored through this information.

[0049] It should be noted that the above-mentioned “sending the image information of the plurality of pixels to the cloud 4” means writing the image information of the pixels constituting the screen in the second area into a file in sequence and then sending it to the cloud 4. That is to say, a series of RGB grayscale values ​​are written into the file, including the image information of all the pixels constituting the second area 211, that is, the upper bottom to the lower bottom of the trapezoidal area, and these image information correspond one-to-one to the pixels constituting the screen in the second area 211, and these pixels are arranged in sequence, for example, from the first pixel in the first row of the second area 211 to the last pixel in the first row, then from the first pixel in the second row to the last pixel in the second row, and so on to the last pixel in the last row.

[0050] The cloud 4 is configured to: S1. obtain the coordinates of the multiple vertices of the second area 211. S2. obtain the third area in the restored image according to the coordinates of the multiple vertices of the second area 211. S3. fill the image information of the multiple pixel points into the multiple pixel points in the third area one by one to obtain the restored image.

[0051] The restored image refers to an image obtained by restoring the image information of the pixel points in the cloud. The restored image is similar to the first image in that the sizes of the two are the same. The difference is that the restored image is restored in the cloud 4, while the first image is captured by the camera. The second area 211 is similar to the third area in that the displayed screen images are the same. The difference is that the screen corresponding to the third area is restored by the coordinates of multiple vertices and the image information of multiple pixel points of the image displayed in the second area 211 transmitted by the oven.

[0052] After the camera captures the first image, the oven does not transmit the image information of all pixels of the first image, but selects the second area in the first image, that is, the area covered with food and can be identified, and transmits the image information of the pixels in the second area and the vertex coordinates of the second area. After receiving the image information of the pixels and the vertex coordinates of the second area, the cloud restores the image. During the whole process, the data to be transmitted is more streamlined and more efficient than transmitting all pixel information to the cloud, and the bandwidth and resources occupied are less and the speed is faster, so that more resources can be allocated to the identification of food and the judgment of the maturity of food. Therefore, the accuracy of intelligent identification in the cloud is also greatly improved.

[0053] As attached Figure 4 and attached Figure 5 As shown, attached Figure 4 and attached Figure 5 FIG. 1 shows the installation position of the photographing device in the oven applicable to the embodiment of the present application. Figure 4 and attached Figure 5 In the figures, the oven 100 is abstracted as a rectangular parallelepiped, and the camera 2 can be arranged on the top and side of the oven. The schematic diagram of the camera 2 being arranged on the top of the oven is shown in the attached figure. Figure 4 As shown, the photographing device 2 is arranged on the side of the oven as shown in the attached diagram. Figure 5 shown.

[0054] Refer to the attached Figure 6 , attached Figure 6 The area that the camera can capture is shown. Figure 6 The arc-shaped area in the image is the area that can be photographed by the camera 2. Since the camera 2 is fixedly installed in the oven 100, the area that can be photographed by the camera 2 is certain, and the size of the first image photographed by the camera 2 is also fixed, the position of the food on the baking tray in the first image is also relatively fixed. That is to say, the position of the second area covered with food and identifiable is fixed relative to the first area of ​​the first image.

[0055] Exemplarily, the processor 3 is configured to extract, in the first image, image information of a plurality of pixels located in the second area 211. Specifically, the processor 3 is configured to connect adjacent coordinates in sequence with line segments according to the coordinates of the vertices of the second area 211 in the first area 21 that are pre-calibrated, obtain the range of the second area 211, and extract image information of a plurality of pixels located in the second area 211.

[0056] For example, see the attached Figure 7 , attached Figure 7 The positional relationship of the second area in the first image is shown. Figure 7 The second area 211 shown in the figure is a trapezoidal area, which is formed by connecting four vertices a, b, c and d in sequence with line segments. Since the position of the second area 211 in the first area 21 is relatively fixed, as long as the relative position coordinates of the four vertices of the second area 211 in the first area 21 are determined, the range of the second area 211 can be obtained by connecting adjacent coordinates in sequence with line segments, and then the image information of multiple pixel points located in the second area 211 can be extracted.

[0057] Combined with Figure 7 For example, the process of pre-calibrating the coordinates of the vertices of the second area in the first area includes: taking the lower left vertex of the first image, i.e. the photo taken by the camera, as the origin, taking the horizontal boundary of the first image as the x-axis, and the vertical boundary as the y-axis, to establish a rectangular coordinate system (referred to as the first coordinate system). Then the coordinates of the four vertices of the second area 211 relative to the first area 21 are: the coordinates of a are (x a ,y a ); the coordinates of b are (x b ,y a ); the coordinates of c are (x c ,y d ); the coordinates of d are (x d ,y d ).

[0058] The processor 3 is configured to determine the positional relationship of the second area relative to the first area based on the four-point coordinates a, b, c, and d, so that in the first image captured by the shooting device, the range of the second area 211 can be obtained according to the pre-calibrated four-point coordinates a, b, c, and d, and the image information of multiple pixel points located in the second area can be extracted.

[0059] It should be noted that the embodiment of the present application takes the second area covered with food and identifiable as an isosceles trapezoid as an example.

[0060] The oven transmits the above coordinates and the image information of multiple pixels in the second area to the cloud. As a possible implementation method, the cloud is configured as follows: S1. Obtaining the coordinates of multiple vertices in the second area, specifically configured as follows:

[0061] S11. Establish a coordinate system (called the second coordinate system). Figure 8 As shown, the second region is a trapezoid, and the multiple vertices of the third region include a first vertex, a second vertex, a third vertex, and a fourth vertex. The first vertex and the second vertex are two endpoints of the upper base of the trapezoid, the third vertex and the fourth vertex are two endpoints of the lower base of the trapezoid, and the second vertex and the third vertex are adjacent. The origin of the coordinate system is the intersection of the perpendicular line of the lower base of the trapezoid and the extension line of the upper base of the trapezoid, and the perpendicular line of the lower base of the trapezoid passes through the fourth vertex. The horizontal axis of the coordinate system is the straight line where the upper base is located, and the vertical axis of the coordinate system is the perpendicular line of the lower base of the trapezoid. The unit length of the coordinate axis is the size of the pixel point.

[0062] S12. Obtain the coordinates (x1, y1) of the first vertex a, the coordinates (x2, y1) of the second vertex b, the coordinates (x3, y2) of the third vertex c, and the coordinates (x4, y2) of the fourth vertex d in the second coordinate system.

[0063] It should be noted that Figure 7 The coordinates of the second region a in the first coordinate system are

[0064] (x a ,y a ); the coordinates of b are (x b ,y a ); the coordinates of c are (x c ,y d ); the coordinates of d are (x d ,y d ) is used to calibrate the position of the second area in the first area. Figure 8 The coordinates of the first vertex (x1, y1), the second vertex (x2, y1), the third vertex (x3, y2), and the fourth vertex (x4, y2) of the second region in the second coordinate system are used to define the relative position relationship between the four vertices of the second region. Since the second coordinate system is a coordinate system determined by the outline of the second region itself, the shape and size of the region defined by the coordinate system of the four vertices in the second coordinate system are unique, whether in the first image or the restored image.

[0065] Since the restored image and the first image may not be the same size, Figure 7 The coordinates of the four vertices in the first coordinate system shown in FIG. 1 are used to obtain the range of the third area in the restored image. Figure 8The range of the third area in the restored image is obtained by using the coordinates of the four vertices in the second coordinate system shown.

[0066] In some embodiments, the cloud is configured to, S2, obtain the third area in the initial restored image according to the coordinates of the plurality of vertices of the second area 211, specifically configured as:

[0067] S21. In the initial restored image, a third coordinate system is established with the upper left vertex of the initial restored image as the origin, the horizontal boundary as the x-axis, and the vertical boundary as the y-axis. Fig. 9 shown.

[0068] S22. In the third rectangular coordinate system, determine the positions of the four vertices (e, f, g, h) of the third area based on the coordinates of the first vertex (x1, y1), the coordinates of the second vertex (x2, y1), the coordinates of the third vertex (x3, y2), and the coordinates of the fourth vertex (x4, y2) in the second coordinate system.

[0069] S23. Use line segments to sequentially connect adjacent vertices of the third area to obtain a range of the third area.

[0070] It should be noted that each pixel of the initial restored image is not filled with image information, for example, the RGB grayscale of each pixel of the initial restored image is 0, that is, the restored image is displayed in black or gray, and the third area is divided in the initial restored image.

[0071] For example, Fig.10 As shown, attached Fig.10 (a) shows the first image captured by the oven camera. Fig.10 (b) shows the initial restored image on the cloud. Fig.10 (c) in the figure shows the restored image on the cloud.

[0072] As a possible implementation, after acquiring the coordinates of the four vertices of the third region, the cloud 4 is configured to fill the image information of the multiple pixels transmitted from the oven into the third region one by one to obtain a restored image. Fig.11 As shown, the cloud is configured as follows: S3: filling the image information of multiple pixels into the third area one by one, and the process of obtaining the restored image is as follows:

[0073] S-100: Get the row number k of the third area.

[0074] S-110: Obtain the coordinates of the starting pixel point of each row in the third area.

[0075] S-120: filling the image information of the plurality of pixels into the plurality of pixels in the third region in sequence according to the coordinates of the starting pixel of each row and the number of pixels included in each row.

[0076] If the cloud wants to fill the third area with the image information of a series of pixels from the file and restore the same picture as that displayed in the second area, it needs to know the position where each pixel in the file should be filled. The image information of the pixels is arranged in order in the file, but since the third area is a trapezoid with a certain height, and the height is much larger than the width of a pixel, this means that if the pixels want to fill the third area, they must not only go through horizontal filling, but also vertical filling.

[0077] For example, as shown in the attached Fig.12 As shown, attached Fig.12 The pixel arrangement constituting the image displayed in the third area is shown. Fig.12 Each rectangle in the image represents a pixel 10. Each rectangle, that is, each pixel 10, has its own RGB grayscale value. All the pixels 10 constitute the third area visible to the naked eye, that is, the surrounding Fig.12 The trapezoidal area shown in .

[0078] When the cloud obtains the number of rows k of the third area, it can know how many rows of pixels 10 need to be filled. When the cloud obtains the coordinates of the starting pixel 10 of each row, since the third area is an isosceles trapezoid, it can calculate the number of pixels 10 that need to be filled in each row. Then, according to the image information in the file, starting from the first row, fill in the number of pixels 10 that need to be filled according to the number of rows.

[0079] For example, in conjunction with Fig.13 , k is equal to 4, the number of pixels 10 needed to fill the first row is 8, the number of pixels 10 needed to fill the second row is 9, the number of pixels 10 needed to fill the third row is 11, and the number of pixels 10 needed to fill the fourth row is 11.

[0080] As a possible implementation method, Fig.13 As shown, for step S-100, the number of rows k in the third area is obtained, and the cloud is specifically configured to: subtract the ordinate of the fourth vertex from the ordinate of the first vertex to obtain the number of rows k, that is, k=y2-y1.

[0081] As a possible implementation method, Fig.14 As shown, for step S-110: obtaining the coordinates of the starting pixel points of each row in the third area. The cloud is configured as:

[0082] The coordinates of the starting pixel of the first row are the coordinates of the first vertex (x1, y1).

[0083] The coordinates of the starting point of the nth row are (x1-[x1*(n-1)) / y2],y1+n-1), where n is 2, 3, 4...k-1, k.

[0084] The coordinates of the starting pixel of the first row are the coordinates of the first vertex (x1, y1). The coordinates of the starting pixels from the second row to the last row need to be calculated. In the third area, the rectangular coordinate system is re-established to obtain the coordinates of the four vertices. Every change of 1 unit in the length of the coordinate axis is equal to the size of a pixel. For example, if the size of a pixel is 1 unit, then the change of the horizontal axis coordinate from x1 to x1+1 is also 1 unit, and the change of the vertical axis coordinate from y1 to y1+1 is also 1 unit.

[0085] Combined with Fig.15 , attached Fig.15 a, e, f, d in the equation are all points, and the distance between point e and point f is p. Take the calculation of the coordinates (x3, y3) of the starting pixel e in the second row as an example: the triangle formed by the origin, the first vertex a, and the fourth vertex d is similar to the triangle formed by the three points a, e, and f, so (x1 / y2) = (p / 1), so p = x1 / y2, so the coordinates of the starting point pixel in the second row are (x1-x1 / y2, y1+1).

[0086] When calculating the coordinates of the starting point pixel of the third row, as shown in the following figure Fig.16 Attached Fig.16 a, h, g, d in the figure are all points, and the distance between h and g is m. To know the coordinates (x4, y4) of the starting point pixel g in the third row, we need to use the triangle formed by the origin, the first vertex a and the fourth vertex d as vertices, which is similar to the triangle formed by the three points a, g and h as vertices. In this case, (x1 / y2) = (p / 2), so p = 2x1 / y2, so the coordinates of the starting point pixel in the third row are (x1-2x1 / y2, y1+2). From this, we can deduce that the coordinates of the starting pixel point in the nth row are (x1-[x1*(n-1)) / y2], y1+n-1), where n is 2, 3, 4...k-1, k.

[0087] As a possible implementation method, Fig.17 As shown, for step S-120: according to the coordinates of the starting pixel point of each row and the number of pixels contained in each row, the image information of the multiple pixels is sequentially filled into the multiple pixels of the third area, and the cloud is configured as follows:

[0088] The number of pixels in the first row is x2-x1;

[0089] The number of pixels in the nth row is 2x1(n-1) / y2.

[0090] The number of pixels in the first row is the difference between the horizontal coordinate of the second vertex and the horizontal coordinate of the first vertex, x2-x1. The number of pixels from the second row to the end of the last row needs to be calculated row by row. According to the above method, the coordinates of the starting pixel of each row can be obtained (x1-[x1*(n-1)) / y2], y1+n-1), where n is 2, 3, 4...k-1, k. Since the third area is an isosceles trapezoid, the coordinates of the ending pixel of the nth row can be obtained by the same logic as (x1+[x1*(n-1)) / y2], y1+n-1), where n is 2, 3, 4...k-1, k. Then the number of pixels in the nth row is the difference between the horizontal coordinate of the ending pixel and the horizontal coordinate of the starting pixel, that is, 2x1(n-1) / y2, where n is 2, 3, 4...k-1, k.

[0091] It should be noted that in actual usage scenarios, the camera in the oven can pre-set a simple food recognition algorithm. If the area where the food is placed is very small, the minimum trapezoid of the food area can be dynamically calibrated to further reduce the number of pixels. Only the pixels in the trapezoidal area where the food is located are extracted, and the speed of transmission to the cloud is further improved.

[0092] For example, as shown in the attached Fig.18 and attached Fig.19 As shown. Fig.18 and attached Fig.19 , attached Fig.18 and attached Fig.19 The trapezoid in the figure represents the baking tray 7, and the circle inside the trapezoid represents the food 6. Fig.18 The food 6 in the image occupies a larger area than the baking tray 7. The minimum recognition of the food in the cloud needs to be completely covered, so the second area can be selected to be the size of the baking tray. Fig.19 In general, the food 6 occupies a smaller area than the baking tray 7. The minimum recognition of the food by the cloud only needs to completely cover the food 6. Therefore, the second area 211 can be selected as Fig.19 The dotted line in the figure indicates the size of the trapezoid.

[0093] The present invention also provides an image processing method, which includes: photographing the internal structure of the cavity of an oven to obtain a first image. The first image includes a first area, the first area includes a second area, and the second area is an area covered with food and can be identified. The first image is received, and in the first image, image information of multiple pixels located in the second area is extracted and the coordinates of multiple vertices of the second area are obtained. The image information of the multiple pixels and the coordinates of the multiple vertices of the second area are sent to the cloud. The coordinates of the multiple vertices of the second area and the image information of the multiple pixels are received. According to the coordinates of the multiple vertices of the second area, a third area in the initial restored image is obtained. The image information of the multiple pixels is filled into the multiple pixels in the third area one by one to obtain a restored image.

[0094] As a possible implementation, in the first image, extracting image information of multiple pixels located in the second area includes: according to the coordinates of the vertices of the second area in the first area that are pre-calibrated, connecting adjacent coordinates in sequence with line segments, obtaining the range of the second area, and extracting image information of multiple pixels located in the second area. The coordinates of the vertices of the pre-calibrated second area in the first area are calibrated in a first coordinate system, and the origin of the first coordinate system is a vertex of the first area.

[0095] As a possible implementation, obtaining the coordinates of multiple vertices of the second area includes: establishing a second coordinate system. The second area is a trapezoid, and the multiple vertices of the second area include a first vertex, a second vertex, a third vertex, and a fourth vertex. The first vertex and the second vertex are two endpoints of the upper base of the trapezoid, and the third vertex and the fourth vertex are two endpoints of the lower base of the trapezoid. The second vertex and the third vertex are adjacent. The origin of the coordinate system is the intersection of the perpendicular line of the lower base of the trapezoid and the extension line of the upper base of the trapezoid, and the perpendicular line of the lower base of the trapezoid passes through the fourth vertex. The horizontal axis of the coordinate system is the straight line where the upper base is located, and the vertical axis of the coordinate system is the perpendicular line of the lower base of the trapezoid. The unit length of the coordinate axis is the size of the pixel. In the second coordinate system, the coordinates of the first vertex (x1, y1), the coordinates of the second vertex (x2, y1), the coordinates of the third vertex (x3, y2), and the coordinates of the fourth vertex (x4, y2) are obtained.

[0096] As a possible implementation, the third region in the initial restored image is obtained according to the coordinates of the plurality of vertices of the second region, including: establishing a third coordinate system in the initial restored image. In the third coordinate system, the positions of the four vertices of the third region are determined according to the coordinates of the first vertex (x1, y1), the coordinates of the second vertex (x2, y1), the coordinates of the third vertex (x3, y2), and the coordinates of the fourth vertex (x4, y2) in the second coordinate system. The range of the third region is obtained according to the four vertices of the third region.

[0097] As a possible implementation, filling the image information of the plurality of pixels into the third region one by one to obtain a restored image includes: obtaining the number of rows k of the third region. Obtaining the coordinates of the starting pixel of each row of the third region. Obtaining the number of pixels contained in each row of the third region. According to the coordinates of the starting pixel of each row and the number of pixels contained in each row, filling the image information of the plurality of pixels into the plurality of pixels of the third region one by one.

[0098] As a possible implementation manner, obtaining the row number k of the third area includes: subtracting the ordinate of the fourth vertex from the ordinate of the first vertex to obtain the row number k.

[0099] As a possible implementation method, the coordinates of the starting pixel point of each row in the third area are obtained, including: the coordinates of the starting pixel point of the first row are the coordinates of the first vertex (x1, y1); the coordinates of the starting point of the nth row are (x1-[x1*(n-1)) / y2], y1+n-1), where n is 2, 3, 4...k-1, k.

[0100] As a possible implementation manner, the number of pixels contained in each row of the third area is obtained, including: the number of pixels in the first row is x2-x1; the number of pixels in the nth row is 2x1(n-1) / y2.

[0101] After the camera captures the first image, the oven does not transmit the image information of all pixels of the first image, but selects the second area in the first image, that is, the area covered with food and can be identified, and transmits the image information of the pixels in the second area and the vertex coordinates of the second area. After receiving the image information of the pixels and the vertex coordinates of the second area, the cloud restores the image. During the whole process, the data to be transmitted is more streamlined and more efficient than transmitting all pixel information to the cloud, and the bandwidth and resources occupied are less and the speed is faster, so that more resources can be allocated to the identification of food and the judgment of the maturity of food. Therefore, the accuracy of intelligent identification in the cloud is also greatly improved.

[0102] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An oven, characterized in that: The oven comprises: Cavity; A photographing device, wherein the photographing device is configured to photograph the internal structure of the oven cavity to obtain a first image, wherein the first image includes a first area, wherein the first area includes a second area, and the second area is an area covered with food and can be identified; A processor configured to receive the first image, extract image information of a plurality of pixels located in a second area in the first image, and obtain coordinates of a plurality of vertices of the second area; and send the image information of the plurality of pixels and the coordinates of the plurality of vertices of the second area to a cloud; The cloud is connected to the oven, and is configured to receive the coordinates of multiple vertices of the second area and image information of multiple pixel points; obtain a third area in an initial restored image based on the coordinates of the multiple vertices of the second area; and fill the image information of the multiple pixel points into the multiple pixel points in the third area one by one to obtain a restored image.

2. The oven according to claim 1, characterized in that: The processor is configured to extract, from the first image, image information of a plurality of pixels located in the second area, and is specifically configured to: According to the coordinates of the vertices of the second area marked in advance in the first area, adjacent coordinates are sequentially connected by line segments to obtain the range of the second area, and image information of a plurality of pixels located in the second area is extracted; The coordinates of the pre-calibrated vertices of the second region in the first region are calibrated in a first coordinate system, and the origin of the first coordinate system is a vertex of the first region.

3. The oven according to claim 1, characterized in that: The processor is configured to obtain coordinates of a plurality of vertices of the second area, and is specifically configured to: A second coordinate system is established in the first image; the second area is a trapezoid, the multiple vertices of the second area include a first vertex, a second vertex, a third vertex and a fourth vertex, the first vertex and the second vertex are two end points of the upper base of the trapezoid, the third vertex and the fourth vertex are two end points of the lower base of the trapezoid, and the second vertex and the third vertex are adjacent; the origin of the coordinate system is the intersection point of the perpendicular line of the lower base of the trapezoid and the extension line of the upper base of the trapezoid, and the perpendicular line of the lower base of the trapezoid passes through the fourth vertex; the horizontal axis of the coordinate system is the straight line where the upper base is located, and the vertical axis of the coordinate system is the perpendicular line of the lower base of the trapezoid; the unit length of the coordinate axis is the size of the pixel point; The coordinates of the first vertex (x1, y1), the coordinates of the second vertex (x2, y1), the coordinates of the third vertex (x3, y2), and the coordinates of the fourth vertex (x4, y2) are obtained in the second coordinate system.

4. The oven according to claim 3, characterized in that: The cloud is configured to obtain a third area in the initial restored image according to the coordinates of the plurality of vertices of the second area, and is specifically configured as follows: Establishing a third coordinate system in the initial restored image; In the third coordinate system, positions of four vertices of the third region are determined according to the coordinates (x1, y1) of the first vertex, the coordinates (x2, y1) of the second vertex, the coordinates (x3, y2) of the third vertex, and the coordinates (x4, y2) of the fourth vertex in the second coordinate system; The range of the third area is obtained according to the four vertices of the third area.

5. The oven according to claim 4, characterized in that: The cloud is configured to fill the image information of the plurality of pixels into the third area one by one to obtain a restored image, including, specifically, being configured as follows: Get the number of rows k in the third region; Get the coordinates of the starting pixel point of each row in the third area; Get the number of pixels contained in each row of the third area; According to the coordinates of the starting pixel point of each row and the number of pixel points contained in each row, the image information of the multiple pixel points is sequentially filled into the multiple pixel points of the third area.

6. The oven according to claim 5, characterized in that: The cloud is configured such that the obtaining of the number of rows k in the third region is specifically configured as follows: Subtract the ordinate of the fourth vertex from the ordinate of the first vertex to get the row number K.

7. The oven according to claim 5, characterized in that: The cloud is configured to obtain the coordinates of the starting pixel points of each row of the third area, specifically configured as follows: The coordinates of the starting pixel of the first row are the coordinates of the first vertex (x1, y1); The coordinates of the starting point of the nth row are (x1-[x1*(n-1)) / y2],y1+n-1), where n is 2, 3, 4...K-1, k.

8. The oven according to claim 5, characterized in that: The cloud is configured to obtain the number of pixels contained in each row of the third area, and is specifically configured as follows: The number of pixels in the first row is x2-x1; The number of pixels in the nth row is 2x1(n-1) / y2.

9. An image processing method, characterized in that: The image processing method comprises: The internal structure of the oven cavity is photographed to obtain a first image; the first image includes a first area, the first area includes a second area, and the second area is an area covered with food and can be identified; Receive the first image, extract image information of a plurality of pixels located in a second area in the first image and obtain coordinates of a plurality of vertices of the second area; and send the image information of the plurality of pixels and the coordinates of the plurality of vertices of the second area to a cloud; Receive the coordinates of multiple vertices and image information of multiple pixel points of the second area; obtain a third area in the initial restored image according to the coordinates of the multiple vertices of the second area; fill the image information of the multiple pixel points into the multiple pixel points in the third area one by one to obtain a restored image.

10. The image processing method according to claim 9, characterized in that: Extracting image information of a plurality of pixels located in a second area in the first image includes: According to the coordinates of the vertices of the second area marked in advance in the first area, adjacent coordinates are sequentially connected by line segments to obtain the range of the second area, and image information of a plurality of pixels located in the second area is extracted; The coordinates of the pre-calibrated vertices of the second region in the first region are calibrated in a first coordinate system, and the origin of the first coordinate system is a vertex of the first region.

Images