Information processing apparatus, control method, and storage medium

Abstract

The invention relates to an information processing apparatus, a control method, and a storage medium. The information processing apparatus includes: a processing unit configured to perform arithmetic processing of a neural network; and a control unit capable of setting, in the processing unit, a first estimation parameter to be applied to a process for photographing control for photographing an image and a second estimation parameter to be applied to a process for processing control of the image. The control unit switches the first inference parameter, which has been set in the processing unit, to the second inference parameter in response to a determination of a focus object.

Description

[0001] (This application is a divisional application of an application filed on July 28, 2020, with application number 2020107388189 and invention title “Information processing device, imaging device, control method and storage medium”.) Technical Field

[0002] The present invention relates to an information processing device, an imaging device, a control method, and a storage medium for performing arithmetic processing of a neural network. Background Art

[0003] An imaging device such as a digital camera performs processing such as recognition of a captured image. In this case, it is conceivable that the imaging device uses a neural network to perform processing such as recognition of a captured image. As a related art, an endoscope system disclosed in Japanese Patent Laid-Open No. 2009-78069 is proposed. In the endoscope system of Japanese Patent Laid-Open No. 2009-78069, command data related to dynamic dictionary update is sent to a capsule endoscope, and the capsule endoscope updates a dynamic dictionary stored in a RAM based on the command data.

[0004] Generally, since the processing of the neural network performs a large number of multiplication-accumulation operations, a dedicated circuit equipped with a multiplication-accumulation operation circuit is used. Here, in the case where multiple neural network processes are performed on the captured image, it is necessary to efficiently use the dedicated circuit for performing the neural network processing. The endoscope system of Japanese Patent Laid-Open No. 2009-78069 updates the dynamic dictionary based on command data, but it cannot efficiently use the dedicated circuit for performing the neural network processing. Summary of the invention

[0005] The present invention provides efficient switching of neural networks applied to images.

[0006] Therefore, the present invention provides an information processing device, comprising: a processing unit, which performs arithmetic processing of a neural network; and a control unit, which is capable of setting a first inference parameter to be applied to a processing used for shooting control for capturing an image and a second inference parameter to be applied to a processing used for processing control of the image in the processing unit, wherein the control unit switches the first inference parameter set in the processing unit to the second inference parameter in response to determination of a focusing object.

[0007] The present invention provides an information processing device, comprising: a plurality of processing units that perform arithmetic processing of a neural network; and a control unit that is capable of setting a first inference parameter to be applied to a process used for shooting control for shooting an image and a second inference parameter to be applied to a process used for processing control of the image in the neural network, wherein the control unit switches the first inference parameter set in one or more of the processing units to the second inference parameter in response to the satisfaction of a predetermined condition.

[0008] The present invention provides an information processing device, comprising: a processing unit, which performs arithmetic processing of a neural network; and a control unit, which is capable of setting a first inference parameter to be applied to a processing used for shooting control for shooting an image and a second inference parameter to be applied to a processing used for processing control of the image in the processing unit, wherein the control unit changes the timing of switching the first inference parameter to the second inference parameter in response to a shooting mode.

[0009] The present invention provides an information processing device, comprising: a processing unit, which performs arithmetic processing of a neural network; and a control unit, which is capable of setting a first inference parameter to be applied to a processing used for shooting control for shooting an image and a second inference parameter to be applied to a processing used for processing control of the image in the processing unit, wherein the control unit switches the first inference parameter set in the processing unit to the second inference parameter before shooting a still image.

[0010] The present invention provides a camera device, comprising: a camera unit that captures an image; a processing unit that performs arithmetic processing of a neural network; and a control unit that is capable of setting a first inference parameter to be applied to a process used for shooting control for capturing the image and a second inference parameter to be applied to a process used for processing control of the image in the processing unit, wherein the control unit switches the first inference parameter set in the processing unit to the second inference parameter in response to determination of a focus object.

[0011] The present invention provides a control method for an information processing device, wherein the information processing device has a processing unit for performing arithmetic processing of a neural network, and the control method includes: in response to determination of a focused object, switching a first inference parameter set in the processing unit to a second inference parameter, wherein the first inference parameter to be applied to a process used for shooting control for shooting an image and the second inference parameter to be applied to a process used for processing control of the image can be set in the processing unit.

[0012] The present invention provides a control method for an information processing device, wherein the information processing device has multiple processing units for performing arithmetic processing of a neural network, and the control method includes: in response to the satisfaction of a predetermined condition, switching a first inference parameter set in one or more of the processing units to a second inference parameter, and the first inference parameter to be applied to the processing used for shooting control for shooting an image and the second inference parameter to be applied to the processing used for processing control of the image can be set in the neural network.

[0013] The present invention provides a control method for an information processing device, wherein the information processing device has a processing unit for performing arithmetic processing of a neural network, and the control method includes: changing the timing of switching a first inference parameter to a second inference parameter in response to a shooting mode, and the first inference parameter to be applied to the processing used for shooting control for shooting an image and the second inference parameter to be applied to the processing used for processing control of the image can be set in the processing unit.

[0014] The present invention provides a control method for an information processing device, wherein the information processing device has a processing unit for performing arithmetic processing of a neural network, and the control method includes: before shooting a still image, switching a first inference parameter set in the processing unit to a second inference parameter, and the first inference parameter to be applied to a process used for shooting control for shooting an image and the second inference parameter to be applied to a process used for processing control of the image can be set in the processing unit.

[0015] The present invention provides a non-temporary computer-readable storage medium storing a computer executable program, wherein the computer executable program executes a control method for an information processing device having a processing unit for performing arithmetic processing of a neural network, the control method comprising: in response to determination of a focused object, switching a first inference parameter set in the processing unit to a second inference parameter, wherein the first inference parameter to be applied to a process used for shooting control for shooting an image and the second inference parameter to be applied to a process used for processing control of the image can be set in the processing unit.

[0016] The present invention provides a non-temporary computer-readable storage medium storing a computer executable program, wherein the computer executable program executes a control method for an information processing device having a plurality of processing units for performing arithmetic processing of a neural network, wherein the control method comprises: in response to satisfaction of a predetermined condition, switching a first inference parameter already set in one or more of the processing units to a second inference parameter, wherein the first inference parameter to be applied to a process used for shooting control for shooting an image and the second inference parameter to be applied to a process used for processing control of the image can be set in the neural network.

[0017] The present invention provides a non-temporary computer-readable storage medium storing a computer executable program, wherein the computer executable program executes a control method for an information processing device having a processing unit for performing arithmetic processing of a neural network, the control method comprising: changing the timing of switching a first inference parameter to a second inference parameter in response to a shooting mode, the first inference parameter to be applied to a process used for shooting control for shooting an image and the second inference parameter to be applied to a process used for processing control of the image being able to be set in the processing unit.

[0018] The present invention provides a non-temporary computer-readable storage medium storing a computer executable program, wherein the computer executable program executes a control method for an information processing device having a processing unit for performing arithmetic processing of a neural network, the control method comprising: before taking a still image, switching a first inference parameter set in the processing unit to a second inference parameter, wherein the first inference parameter to be applied to a process used for shooting control for taking an image and the second inference parameter to be applied to a process used for processing control of the image can be set in the processing unit.

[0019] According to the present invention, the neural network applied to the image can be efficiently switched.

[0020] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a block diagram showing an imaging apparatus.

[0022] Figure 2 is a diagram showing the structure of a neural network processing unit according to the first embodiment.

[0023] Figure 3 is a flowchart showing the overall processing flow.

[0024] Figure 4: is a flowchart showing the flow of still image shooting processing.

[0025] Figure 5 : is a flowchart showing the flow of continuous shooting processing.

[0026] Figure 6 : is a flowchart showing the flow of moving image shooting processing.

[0027] Fig. 7A is an example of a menu screen, and Figure 7B This is an example of a selection screen.

[0028] Figure 8 is a diagram showing the structure of a neural network processing unit according to the second embodiment.

[0029] Fig. 9 is a flowchart showing the flow of still image capturing processing according to the second embodiment.

[0030] Fig.10 : is a flowchart showing the flow of continuous shooting processing according to the second embodiment.

[0031] Fig.11 is a flowchart showing the flow of moving image shooting processing according to the second embodiment. DETAILED DESCRIPTION

[0032] First embodiment

[0033] Figure 1 1 is a block diagram showing an imaging device 100. The imaging device 100 has a CPU 101, a memory 102, a nonvolatile memory 103, an operation unit 104, and a neural network processing unit 105. The imaging device 100 also has a shooting lens 111, an imaging unit 112, an image processing unit 113, an encoding processing unit 114, a display control unit 115, and a display unit 116. The imaging device 100 also has a communication control unit 117, a communication unit 118, a recording medium control unit 119, and an internal bus 130. The imaging device 100 forms an optical image of a subject on a pixel array of the imaging unit 112 by using the shooting lens 111. The shooting lens 111 may be removable or may be non-removable relative to the body (housing or body) of the imaging device 100. The imaging device 100 writes and reads image data relative to the recording medium 120 via the recording medium control unit 119. The recording medium 120 may also be removable or non-removable with respect to the imaging apparatus 100 .

[0034] The CPU 101 (first CPU) controls the operation of each unit of the camera apparatus 100 via the internal bus 130 by executing a computer program stored in the nonvolatile memory 103. The CPU 101 corresponds to a control unit or a first processor. The memory 102 is a rewritable volatile memory. The memory 102 temporarily stores information such as a computer program for controlling the operation of each unit of the camera apparatus 100 and parameters related to the operation of each unit of the camera apparatus 100, and information received by the communication control unit 117. In addition, the memory 102 temporarily stores an image (image data) acquired by the camera unit 112, or an image and information processed by the image processing unit 113 or the encoding processing unit 114. The memory 102 has a storage capacity for storing various information. The memory 102 stores information used by the neural network processing unit 105. The information used by the neural network processing unit 105 includes a computer program describing the processing content of the neural network, and coefficient parameters (weight coefficients and bias values, etc.) of machine learning. The weight coefficient is a value indicating the strength of the connection between nodes in the neural network. Bias is a value that provides an offset to the integral value of the weight coefficient and the input data. The value of the coefficient parameter of machine learning is adjusted for the neural network through machine learning.

[0035] The nonvolatile memory 103 is a memory capable of electrically erasing and storing. For example, an EEPROM or a hard disk is used as the nonvolatile memory 103. The nonvolatile memory 103 stores information such as a computer program executed by the CPU 101 and parameters related to the operation of each unit of the imaging device 100. When the CPU 101 executes the computer program, various operations performed by the imaging device 100 are realized. The information used by the above-mentioned neural network can be stored in the nonvolatile memory 103.

[0036] The operation unit 104 is provided for the user to operate the imaging apparatus 100. The operation unit 104 includes various buttons such as a power button, a menu button, a release button for shooting, a moving image recording button, and a cancel button. The various buttons may be constituted by switches and a touch screen, etc. The CPU 101 controls the imaging apparatus 100 according to instructions from the user input via the operation unit 104. The CPU 101 may control the imaging apparatus 100 based on a request input from a remote controller or a mobile terminal, for example, via the communication unit 118. The neural network processing unit 105 will be described later.

[0037] The shooting lens 111 is a lens unit configured to include a lens group, a lens controller, and a diaphragm. The lens group includes a zoom lens and a focus lens. The shooting lens 111 can be used as a zoom lens for changing the viewing angle. The lens controller adjusts the focus and controls the aperture value (F value) according to the control signal sent by the CPU 101. The imaging unit 112 sequentially acquires a plurality of images constituting a moving image. For example, a surface image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) device is applied to the imaging unit 112. The imaging unit 112 has a pixel array in which a photoelectric converter (not shown) that converts an optical image of a subject into an electrical signal is arranged in a matrix (i.e., two-dimensionally). An optical image of a subject is formed on the pixel array from the shooting lens 111. The imaging unit 112 outputs the captured image to the image processing unit 113 or the memory 102. The imaging unit 112 can also acquire an image as a still image.

[0038] The image processing unit 113 performs predetermined image processing on the image data that has been output from the imaging unit 112 or the image data that has been read from the memory 102. As the image processing, dynamic range conversion processing, interpolation processing, reduction processing (resizing processing), color conversion processing, and the like can be applied. In addition, the image processing unit 113 uses the image acquired by the imaging unit 112 to perform predetermined arithmetic processing for performing exposure control and distance measurement control, etc. The CPU 101 performs exposure control and distance measurement control, etc. based on the result of the arithmetic processing performed by the image processing unit 113. For example, the CPU 101 performs automatic exposure (AE) processing, automatic white balance (AWB) processing, automatic focus (AF) processing, and the like.

[0039] The encoding processing unit 114 performs intra-frame prediction encoding (intra-screen prediction encoding) and inter-frame prediction encoding (inter-screen prediction encoding) on ​​the image data. Thus, the size of the image data is compressed. The encoding processing unit 114 may be, for example, an encoding device composed of a semiconductor device or the like. In addition, the encoding processing unit 114 may be an encoding device provided outside the camera 100, and the camera 100 may obtain the encoded image data from the external encoding device.

[0040] The display control unit 115 controls the display unit 116. The display control unit 115 is implemented by a GPU, for example. The display control unit 115 generates an image that can be displayed on the display screen of the display unit 116, and outputs the generated image to the display unit 116 as an image signal. In addition, the display control unit 115 can not only output image data to the display unit 116, but also output the image data to the external device 217 via the communication control unit 117. The display unit 116 displays an image on the display screen based on the image signal sent from the display control unit 115. The display unit 116 has an on-screen display (OSD) function for displaying a setting screen such as a menu on the display screen. The display control unit 115 can output the image signal to the display unit 116 in a state where the OSD image is superimposed on the image signal. The display unit 116 is composed of a liquid crystal display or an organic EL display, etc., and displays the image signal sent from the display control unit 115. The display unit 116 can also be a touch screen display, for example. In the case where the display unit 116 is a touch screen display, the display unit 116 also has the function of the operation unit 104.

[0041] The communication control unit 117 is controlled by the CPU 101. The communication control unit 117 controls the communication unit 118. The communication control unit 117 sends a video signal that complies with a communication standard such as a High Definition Multimedia Interface (HDMI) (registered trademark) or a serial digital interface (SDI). In addition, the communication unit 118 is also capable of sending and receiving control signals other than video signals. The communication unit 118 converts the video signal and the control signal into physical electrical signals, and sends and receives these physical electrical signals with respect to the external device 217. The communication control unit 117 can generate a modulated signal that complies with a predetermined wireless communication standard, and send the modulated signal from the communication unit 118. In addition, the communication control unit 117 can also obtain the modulated signal from the external device 217 received by the communication unit 118.

[0042] The recording medium control unit 119 controls the recording medium 120. The recording medium control unit 119 outputs a control signal for controlling the recording medium 120 to the recording medium 120 based on a request from the CPU 101. As the recording medium 120, for example, a nonvolatile memory, a magnetic disk, or a semiconductor memory is applied. The recording medium 120 stores encoded image data and the like. In the recording medium 120, the image data and the like are saved as a file in a form conforming to the file system of the recording medium 120. Each unit such as the CPU 101 and the neural network processing unit 105 is accessible to each other via the internal bus 130.

[0043] Figure 21 is a diagram showing the structure of the neural network processing unit 105. The neural network processing unit 105 is constructed by a neural core 140 as a processing unit or a second processor. The neural core 140 performs inference processing by performing arithmetic processing using pre-learned coefficient parameters. The neural network (neural network system) of each embodiment will be described as a convolutional neural network (CNN). However, neural networks other than CNN can be applied. As a CNN, for example, a layer structure in which convolution layers and pooling layers are alternately stacked and a fully connected layer is connected to the output side can be adopted. The coefficient parameters of the above-mentioned machine learning correspond to the weights or biases possessed by the respective edges connected between the nodes of each layer in the fully connected layer. In addition, the coefficient parameters of machine learning correspond to the weights or biases of the kernel (filter) in each layer of the previous level of the fully connected layer.

[0044] The neural core 140 is, for example, a dedicated circuit for processing a neural network. The neural core 140 includes a CPU 141, a multiplication and addition operation circuit 142, a DMA 143, and an internal memory 144. The CPU 141 (second CPU) is different from Figure 1 The CPU 141 is a CPU of the CPU 101 in the embodiment. The CPU 141 executes a computer program describing the processing content of the neural network. The computer program can be read by the CPU 141 from the memory 102 or the nonvolatile memory 103 via the internal bus 130, or can be read from the internal memory 144. In addition, the CPU 141 also controls the multiplication and addition operation circuit 142 and the DMA 143.

[0045] The multiplication-accumulation operation circuit 142 is a circuit that performs multiplication-accumulation operation in the processing of the neural network. The multiplication-accumulation operation circuit 142 is composed of a plurality of multiplication-accumulation operation circuits, and each multiplication-accumulation operation circuit can perform arithmetic processing in parallel. The DMA 143 is a circuit that performs data transmission without passing through the CPU 141. The DMA 143 controls data transmission between the memory 102 or the non-volatile memory 103 and the internal memory 144 via the internal bus 130. The DMA 143 also controls data transmission between the multiplication-accumulation operation circuit 142 and the internal memory 144. The data transmitted by the DMA 143 includes a computer program that describes the processing content of the neural network, coefficient parameters of machine learning, and intermediate data calculated by the multiplication-accumulation operation circuit 142.

[0046] The internal memory 144 stores a computer program describing the processing contents of the neural network, coefficient parameters of machine learning, and intermediate data calculated by the multiplication and accumulation circuit 142. In addition, the internal memory 144 may have a plurality of blocks. In this case, each block may be switched dynamically.

[0047] Next, the normal operation performed by the camera device 100 of the present embodiment will be described. For example, when the user operates the power button of the operation unit 104, the operation unit 104 outputs a start instruction to the CPU 101 based on the operation. Upon receiving the instruction, the CPU 101 controls the power supply section to supply power to each block of the camera device 100. When supplying power to each block, for example, the CPU 101 confirms the mode switching switch of the operation unit 104. The modes of the present embodiment include a still image shooting mode, a continuous shooting mode, a moving image shooting mode, and a reproduction mode. The mode switching switch is a switch for setting any of these modes. The user can switch to any of the four modes by switching the mode switching switch. The switchable modes are not limited to the above four modes. Based on the instruction signal from the operation unit 104, the CPU 101 confirms the mode set by the mode switching switch.

[0048] A normal still image shooting mode will be described. When the user half-presses the release button of the operation unit 104 in the shooting standby state, the camera apparatus 100 is focused. Then, when the user fully presses the release button of the operation unit 104, the camera apparatus 100 performs a recording process of the still image that has been shot. In the recording process, the image processing unit 113 performs image processing on the image data of the still image shot by the camera unit 112. Then, the encoding processing unit 114 performs encoding processing on the image data that has undergone the image processing, and the recording medium control unit 119 records the encoded image data as an image file in the recording medium 120.

[0049] In the present embodiment, the imaging device 100 has a function of continuously shooting still images. When the user sets a shooting mode for performing continuous shooting by using the operation unit 104, the continuous shooting function becomes effective. Information indicating whether the continuous shooting function is effective or not is stored in the memory 102. While the release button is fully pressed in a state in which the continuous shooting function is effective, the imaging device 100 continuously shoots still images. The mode in which the continuous shooting function is effective is the continuous shooting mode.

[0050] When the user presses the moving image shooting button of the operation unit 104 in the moving image shooting mode, the camera 100 shoots a moving image. In the moving image shooting mode, the image processing unit 113 performs image processing on the continuous moving image data shot by the camera unit 112, and the encoding processing unit 114 performs encoding processing on the image-processed moving image data as a moving image. Then, the recording medium control unit 119 records the encoded moving image data as a file in the recording medium 120.

[0051] In the shooting standby state, the camera unit 112 shoots an image at a predetermined frame rate, and the image processing unit 113 performs image processing for display on the image. Then, the display control unit 115 causes the display unit 116 to display the image data that has undergone the image processing for display. Thus, a live view image is displayed. In the reproduction mode, the recording medium control unit 119 reads an image file stored in the recording medium 120, and the encoding processing unit 114 decodes the image data of the read image file. Then, the image processing unit 113 performs processing for display on the image data, and the display control unit 115 causes the display unit 116 to display the image data.

[0052] Next, the overall processing performed by the neural network processing unit 105 will be described. Figure 3 1 is a flowchart showing the overall processing flow performed by the neural network processing unit 105. When the user performs an operation to turn on the power of the imaging apparatus 100, the power of the imaging apparatus 100 is turned on. Then, the computer program stored in the nonvolatile memory 103 is expanded in the memory 102, and the CPU 101 reads and executes the computer program expanded in the memory 102. The CPU 101 causes the imaging unit 112 to capture images at a predetermined frame rate, and causes the display unit 116 to start displaying a live view image.

[0053] The CPU 101 determines whether the mode switching switch of the operation unit 104 is set to the motion image shooting mode (S301). When it is determined as "No" in S301, the CPU 101 determines whether the continuous shooting mode is set (S302). The CPU 101 makes the determination of S302 based on the information stored in the memory 102 regarding whether the continuous shooting function is valid or invalid. When it is determined as "No" in S302, the CPU 101 controls to execute the still image shooting process (S303). When it is determined as "Yes" in S302, the CPU 101 controls to execute the continuous shooting process (S304). When it is determined as "Yes" in S301, the CPU 101 controls to execute the motion image shooting (S305). The details of the still image shooting process, the continuous shooting process, and the motion image shooting process will be described later.

[0054] Next, the still image capturing process of S303 will be described. Figure 41 is a flowchart showing the flow of still image shooting processing. CPU 101 notifies CPU 141 of the first inference parameter read from memory 102. Upon receiving the notification, CPU 141 controls DMA 143 to read the first inference parameter (S401). The read first inference parameter is stored in internal memory 144. Thus, the first inference parameter is set in neural core 140. The first inference parameter is a coefficient parameter of machine learning used when applying a neural network to shooting control.

[0055] CPU 101 notifies CPU 141 that processing of a neural network using a first inference parameter is performed. With a live view image as input, CPU 141 causes the multiplication-addition operation circuit 142 to perform multiplication-addition operation processing, and performs processing of a neural network to which the first inference parameter is applied. Thus, image analysis processing using the first inference parameter is performed as processing for shooting control (S402). Analysis processing using the first inference parameter includes determination of a shooting mode and detection of a subject (human body and object). The processing used for shooting control using the first inference parameter may be other processing for setting shooting parameters. By performing S402, the shooting mode of a still image such as a portrait mode or a landscape mode is determined, and the subject is detected. The processing result of S402 is displayed on the display unit 116.

[0056] Next, the CPU 101 determines whether the release button of the operation unit 104 is half-pressed and whether the focus object is determined (S403). When it is judged as "Yes" in S403, the camera device 100 has received a shooting preparation instruction. In this case, the process goes to S405 described later. When it is judged as "No" in S403, the CPU 101 determines whether the focus object is determined (S404) even when the release button is not half-pressed. For example, in the case where the display unit 116 is a touch screen display, the focus object can be determined when the user touches the display unit 116. Optionally, the focus object can be automatically determined by analysis processing using the first inference parameter. When it is judged as "No" in S404, the process returns to S402.

[0057] In the case of "yes" in S403 or "yes" in S404, CPU 101 notifies CPU 141 of the second inference parameter read from memory 102. Upon receiving the notification, CPU 141 controls DMA 143 to read the second inference parameter (S405). The read second inference parameter is stored in internal memory 144. Therefore, the inference parameter set in neural core 140 is switched from the first inference parameter to the second inference parameter. The second inference parameter is a coefficient parameter of machine learning used when applying a neural network to image processing control.

[0058] The CPU 101 determines whether the release button of the operation unit 104 is fully pressed (S406). When it is judged as "No" in S406, it is determined whether the release button is released (whether the half-press is released), or whether the determination of the focus object is released (S407). It should be noted that in the case where the release button is released, or the determination of the focus object is released, the process returns to S401. In this case, the inference parameter set in the neural core 140 is switched from the second inference parameter to the first inference parameter in S401. In the case where the release button is not released and the determination of the focus object is not released, the process returns to S406, in which the release button of the operation unit 104 is fully pressed, and the process waits until shooting is performed.

[0059] In the case where the judgment is "yes" in S406, the CPU 101 captures a still image (S408). In addition, the CPU 101 notifies the CPU 141 of the second inference parameter read from the memory 102. With the image captured in S408 as input, the CPU 141 causes the multiplication and addition operation circuit 142 to perform a multiplication and addition operation process to perform a neural network process to which the second inference parameter is applied. Thus, a neural network process using the second inference parameter is performed as a process used for image processing control (S409). The neural network process using the second inference parameter includes image filtering processing, image restoration processing, super-resolution processing, development processing, and processing of a part of the encoding processing unit 114. By performing S407, for example, image restoration processing of the image is performed. The processing result of the neural network process using the second inference parameter can be output to the recording medium control unit 119. Thus, the image that has undergone the development process is recorded on the recording medium 120.

[0060] In this way, before the focus object is determined, the first inference parameter is set in the neural core 140. Then, the neural core 140 performs a neural network process using the first inference parameter. Then, in response to the focus object being determined, the inference parameter set in the neural core 140 is switched from the first inference parameter to the second inference parameter. Thus, the neural core 140 can perform a neural network process using the second inference parameter. Here, after the release button of the operation unit 104 is fully pressed, the inference parameter set in the neural core 140 is switched from the first inference parameter to the second inference parameter. In this case, the process used for switching takes time, and the start of the neural network process using the second inference parameter is delayed. Generally, after the focus object used for still image shooting is determined, the shooting mode or the focus object subject is unlikely to change. Therefore, even in the case where the inference parameter set in the neural core 140 is switched to the second inference parameter in response to the focus object being determined, the influence on the shooting control is small. In this way, the neural network process using the second inference parameter can be started immediately after the still image is shot. As described above, the neural network applied to the image can be efficiently switched according to the flowchart used for still image shooting.

[0061] Next, we will explain Figure 3 The continuous shooting process in S304. Figure 5 : is a flowchart showing the flow of continuous shooting processing. The continuous shooting processing is performed when the continuous shooting mode is set. First, the neural core 140 reads the first inference parameter in response to the notification from the CPU 101 (S501). The processing of S501 is the same as the processing of S401. In addition, the neural core 140 performs a neural network processing using the first inference parameter as a processing for shooting control (S502). The processing of S502 is the same as the processing of S402. In the continuous shooting processing, the neural network processing using the first inference parameter continues even when the release button of the operation unit 104 is half-pressed.

[0062] The CPU 101 determines whether the release button of the operation unit 104 is fully pressed (S503). In a case where the release button is not fully pressed (in the case of NO in S503), the flow returns to S502.

[0063] In the case of YES in S503 , the CPU 101 captures a still image ( S504 ). It should be noted that in the continuous shooting mode, all images captured by the imaging apparatus 100 are temporarily stored in the memory 102 or the recording medium 120 .

[0064] In addition, the CPU 101 determines whether the continuous shooting is completed (S505). The CPU 101 determines that the user has instructed to complete the continuous shooting if the release button of the operation unit 104 is fully pressed, and determines that the continuous shooting is continued if the release button of the operation unit 104 is not released.

[0065] If it is determined as "NO" in S505, after performing the neural network processing using the first estimation parameter (S506), the next still image shooting is performed (S504).

[0066] In the case where the judgment is "yes" in S505, the CPU 101 notifies the neural core 140 of the second inference parameter. Then, the neural core 140 reads the second inference parameter (S507). The processing of S507 is the same as the processing of S405. Therefore, the inference parameter set in the neural core 140 is switched from the first inference parameter to the second inference parameter. Then, the neural core 140 performs the neural network processing using the second inference parameter as the processing used for image processing control (S508). The processing of S508 is the same as the processing of S409. It should be noted that in the continuous shooting process, the CPU 101 erases the still images that have been processed by S508 from the memory 102 or the recording medium 120. When the CPU 101 performs the processing of S508 on all still images temporarily stored in the memory 102 or the recording medium 120, the processing ends.

[0067] In the case where the continuous shooting mode is set, until the continuous shooting ends, the neural core 140 performs neural network processing using the first inference parameter as the processing used for shooting control on multiple still images to be continuously shot. Then, in response to the end of the continuous shooting, the first inference parameter set in the neural core 140 is switched to the second inference parameter. The neural core 140 performs neural network processing on the still images using the second inference parameter. In the case where the subject is a moving object, it is desired to detect the subject by using continuously shot still images or live view images between still images so that the camera device 100 does not lose sight of the position of the subject. Therefore, in the continuous shooting mode, even if the start of the neural network processing using the second inference parameter is slightly delayed, the first inference parameter is set in the neural core 140 until the continuous shooting ends.

[0068] Next, we will explain Figure 3 The moving image capturing process in S305. Figure 6: is a flowchart showing the flow of motion image shooting processing. The CPU 101 determines which of the processes used for shooting control and image processing control the neural core 140 is to perform (S601). The determination of S601 can be performed according to the user's selection. Although the details of the determination of S601 will be described later, if the user selects shooting priority as the AI ​​priority mode, it is determined that the process used for shooting control is selected, and when the user selects recording priority as the AI ​​priority mode, it is determined that the process used for image processing control is selected. When it is determined in S601 that "the process used for shooting control" is selected, the CPU 101 notifies the neural core 140 of the first inference parameter. Then, the neural core 140 reads the first inference parameter (S602). The neural core 140 performs a neural network process using the first inference parameter as the process used for shooting control (S603). The process of S603 is the same as the process of S402. The CPU 101 determines whether motion image shooting has started (S604).

[0069] If it is determined as "No" in S604, the flow returns to S603. If it is determined as "Yes" in S604, the neural core 140 performs a neural network process using the first estimation parameter as a process for shooting control during moving image shooting (S605).

[0070] The CPU 101 determines whether the moving image shooting has ended (S606). Whether the shooting has ended can be determined based on whether the shooting using the moving image shooting button of the operation unit 104 is continued. When it is determined as "No" in S606, the flow returns to S605. When it is determined as "Yes" in S606, the processing ends.

[0071] If it is determined in S601 that "processing for image processing control" is selected, the CPU 101 notifies the neural core 140 of the second estimation parameter. Then, the neural core 140 reads the second estimation parameter (S607). The CPU 101 determines whether moving image shooting has started (S608).

[0072] When it is determined as "No" in S608, the process waits until the determination becomes "Yes", and when it is determined as "Yes" in S608, the process proceeds to S609. The neural core 140 performs a neural network process using the second inference parameter as a process for image processing control during motion image shooting (S609). The process of S609 is the same as the process of S409.

[0073] The CPU 101 determines whether the moving image shooting has ended (S610). The determination of S610 is the same as the determination of S606. When the determination in S610 is "No", the flow returns to S609. When the determination in S610 is "Yes", the process ends.

[0074] As described above, in the motion image shooting process, the processing of the neural network using the first inference parameter or the processing of the neural network using the second inference parameter is performed. Since the amount of computation of the multiplication and addition operation in the processing of the neural network is large, the processing of the neural network for one image requires a certain amount of time. On the other hand, in motion image shooting, images are acquired sequentially according to the frame rate. Therefore, it is difficult for one neural core 140 to apply both the processing of the neural network using the first inference parameter and the processing of the neural network using the second inference parameter to each image acquired at a high speed. Therefore, when the motion image shooting mode is set, it is up to the user or the like to select whether to apply the neural network processing to the pre-exposure processing or the neural network processing to the post-exposure processing.

[0075] Fig. 7A and 7B is a diagram showing an example of a screen for selecting a priority mode. Fig. 7A 201 is a menu screen as a screen displayed on the display unit 116. The menu screen has a plurality of items. By operating the operation unit 104 and moving the cursor 201, the user can select an item for changing the setting. In the case where the display unit 116 is a touch screen display, the user can perform a selection operation on the display unit 116. Assume that Fig. 7A In response to this operation, the display control unit 115 causes the display unit 116 to display Figure 7B The selection screen is shown.

[0076] Figure 7B The selection screen is a screen for selecting one of "shooting priority" and "recording priority". By operating the cursor 202 on the operation unit 104 or the display unit 116, the user can select one of "shooting priority" and "recording priority". "Shooting priority" is an item selected when applying neural network processing to shooting control. "Recording priority" is an item selected when applying neural network processing to image processing control. When the setting button 203 is pressed in a state where any of these items is selected, the CPU 101 recognizes to which of the processing used for shooting control and the processing used for image processing control the neural network processing is applied. Thus, the CPU 101 can make a judgment at S601. The judgment at S601 can be made by excluding the use of Fig. 7A and Figure 7B For example, when there is no instruction from the user, the CPU 101 may determine that the neural network processing is applied to the processing for shooting control as the initial value. In addition, the CPU 101 may determine that the neural network processing is applied to the processing for shooting control and the processing for image processing control, whichever is the last instructed by the user.

[0077] In addition, the amount of computation of the neural network processing using the first inference parameter applied when shooting moving images can be smaller than the amount of computation of the neural network processing using the first inference parameter applied when shooting still images. In this case, the first inference parameter used for shooting moving images and the first inference parameter used for shooting still images are stored in the memory 102 or the recording medium 120. As described above, in shooting moving images, images are acquired at a predetermined frame rate. For this reason, when shooting moving images, it is necessary to perform neural network processing on the acquired images at high speed. Here, if the amount of computation of the first inference parameter when shooting moving images is small, the neural network processing using the first inference parameter can be performed on the image acquired at a high frame rate by shooting moving images. The second inference parameter is also the same as the first inference parameter. That is, the amount of computation of the second inference parameter applied when shooting moving images can be smaller than the amount of computation of the second inference parameter applied when shooting still images.

[0078] In addition, the CPU 101 may select a second inference parameter from the plurality of second inference parameters based on the processing result of the neural network using the first inference parameter. In this case, the CPU 101 may select the most appropriate second inference parameter from the second inference parameters based on the processing result of the neural network using the first inference parameter.

[0079] For example, assume that a human body is detected as a subject by neural network processing using a first inference parameter, and portrait mode is determined as a shooting mode. In this case, CPU 101 selects an inference parameter used for a filter process or an image restoration process that makes the subject soft and bright as a second inference parameter. Neural core 140 performs a neural network process using the selected inference parameter. On the other hand, assume that a human body and a specific object are not detected by neural network processing using the first inference parameter, and it is determined to be a landscape mode. In this case, CPU 101 selects an inference parameter used for a filter process or an image restoration process that emphasizes the contrast and saturation of the subject as a second inference parameter.

[0080] In addition, the memory 102 or the recording medium 120 can store a third inference parameter in addition to the first inference parameter and the second inference parameter of multiple types. The third inference parameter is a coefficient parameter of machine learning used when applying a neural network to reproduce a still image or a moving image. The CPU 101 can also select a third inference parameter from multiple types of third inference parameters based on the processing result of the neural network processing using the first inference parameter. The reproduction processing includes, for example, filtering processing of the reproduced image and processing of detecting a specific subject (object or person) from the reproduced image. In addition, the third inference parameter can also be used when automatically displaying a preview image after shooting is completed.

[0081] As described above, according to the present embodiment, the inference parameters to be set in the neural core 140 are dynamically switched at a timing suitable for each imaging mode. Thus, the neural network to be set in the neural core 140 can be efficiently switched according to the situation.

[0082] Second embodiment

[0083] Since the configuration of the imaging apparatus 100 is the same as that of the first embodiment, the description thereof will be omitted. Figure 8 1 is a diagram showing the structure of the neural network processing unit 105 according to the second embodiment. The neural network processing unit 105 has a neural core 140 and a neural core 150. The neural core 140 has a CPU 141, a multiplication-accumulation operation circuit 142, a DMA 143, and an internal memory 144. The neural core 150 has a CPU 151, a multiplication-accumulation operation circuit 152, a DMA 153, and an internal memory 154.

[0084] The neural network processing unit 105 may also have three or more neural cores 140. Figure 8 In the example of , each neural core has an internal memory, but the neural network processing unit 105 may also have an internal memory used by each neural core. In addition, in the second embodiment, the CPU 101 or the GPU of the display control unit 115 may also be used to process the neural network. In the second embodiment, each inference parameter is divided into a plurality of processing units. The processing content varies depending on the processing unit.

[0085] For example, it is assumed that the first inference parameter used for shooting control is an inference parameter applied to the processing of the neural network used for subject detection and shooting mode judgment. In this case, the first inference parameter includes the first inference parameter used for subject detection and the first inference parameter used for shooting mode judgment as processing units. It is also assumed that the second inference parameter used for image processing control is an inference parameter applied to image restoration processing and encoding processing. In this case, the second inference parameter includes the second inference parameter used for image restoration processing and the second inference parameter used for encoding processing as processing units.

[0086] Reference Fig. 91 is used to explain the overall processing flow performed by the neural network processing unit 105 in the second embodiment. The CPU 101 notifies the CPU 141 and the CPU 151 of the first inference parameter read from the memory 102 for each processing unit. Upon receiving the notification, the CPU 141 controls the DMA 143 to read the first inference parameter used for subject detection, and the CPU 151 controls the DMA 153 to read the first inference parameter used for shooting mode determination (S901).

[0087] The CPU 101 notifies the CPU 141 and the CPU 151 to execute the process of the neural network using the first inference parameter. With the live view image as input, the CPU 141 and the CPU 151 execute the process of the neural network to which the first inference parameter is applied (S902).

[0088] Next, the CPU 101 determines whether the release button of the operation unit 104 is half-pressed and the focus object is determined (S903). When it is determined as "yes" in S903, the process goes to S905. When it is determined as "no" in S903, even when the release button is not half-pressed, the CPU 101 determines whether the focus object is determined (S904). When it is determined as "no" in S904, the process returns to S902.

[0089] When it is judged as "yes" in S903 or "yes" in S904, CPU 101 only notifies CPU 151 of the second inference parameter read from memory 102. Upon receiving the notification, CPU 151 controls DMA 153 to read the second inference parameter used for encoding processing (S905). The read second inference parameter is stored in internal memory 154. Thus, the inference parameter set in neural core 150 is switched from the first inference parameter to the second inference parameter. At this time, the inference parameter set in neural core 140 remains the first inference parameter.

[0090] With the live view image as input, the CPU 141 performs a process of the neural network to which the first estimation parameter is applied ( S906 ).

[0091] CPU 101 determines whether the release button of operation unit 104 is fully pressed (S907). When it is judged as "No" in S907, CPU 101 determines whether the release button is released (whether the half-press is released), or whether the determination of the focus object is released (S908). In the case where the release button is released or the determination of the focus object is released, CPU 101 causes CPU 151 to read the first inference parameter used for the determination of the shooting mode (S909). Then, the processing flow returns to S902. In the case where the release button is released or the determination of the focus object is released, it is unlikely that a still image can be taken immediately, so shooting control using the first inference parameter can be performed by two neural cores. On the contrary, when it is judged as "No" in S908, the process returns to S906.

[0092] When the judgment is "yes" in S907, the CPU 101 captures a still image (S910). Furthermore, the CPU 151 performs a neural network process to which the second inference parameter is applied, using the image captured in S910 as an input. Thus, the neural network process using the second inference parameter is performed as a process for image processing control (S911).

[0093] In this way, before determining the focus object, the first inference parameter is set in both the neural core 140 and the neural core 150. Then, in response to the determination of the focus object as a predetermined condition, only the inference parameter that has been set in the neural core 150 is switched from the first inference parameter to the second inference parameter. Therefore, the neural core 150 can perform the neural network processing using the second inference parameter. Thus, the neural network processing using the second inference parameter can be started immediately after taking a still image.

[0094] Here, the inference parameter set in the neural core 140 is maintained as the first inference parameter. Therefore, the processing speed becomes lower than the processing speed before the focus object is determined, but after the focus object is determined, the neural network processing using the first inference parameter can be performed with the live view image as an input. It should be noted that although the first inference parameter is notified to the CPU 141 and the CPU 151 for each processing unit separately here, the same first inference parameter can be notified to the CPU 141 and the CPU 151.

[0095] After S905, the CPU 101 may cause the display unit 116 to display information indicating that the processing speed of the neural network using the first inference parameter has been decreasing. As described above, at the stage of S902, the two neural cores, the neural core 140 and the neural core 150, perform neural network processing using the first inference parameter. On the other hand, at the stage of S906, the neural core 150 performs neural network processing using the first inference parameter, but the neural core 150 switches to the second inference parameter and waits until a still image is captured. Therefore, at the stage of S906, the processing speed of the neural network using the first inference parameter slows down compared to at the stage of S902. The display of the above information makes it less likely that the user will feel uncomfortable about the slowdown in the speed of subject detection and shooting mode judgment.

[0096] It should be noted that Fig. 9 The flowchart can also be used in continuous shooting processing. Fig.10 is a flowchart of the continuous shooting process. Fig.10 In the continuous shooting process of Fig. 9 The same processing as the still image shooting processing of the image capture apparatus 100 is performed until S910. In the case where the judgment is "yes" in S907 and the CPU 101 shoots a still image in S910, the CPU 101 judges whether the continuous shooting has been completed (S1011). It should be noted that all still images shot by the image capture apparatus 100 are temporarily stored in the memory 102 or the recording medium 120.

[0097] When it is determined as NO in S1011, the CPU 141 uses the still image or the live view image obtained after the still image as input, performs the processing of the neural network to which the first inference parameter is applied (S1012), and then performs the next still image shooting (S910).

[0098] When the judgment is "yes" in S1011, the CPU 151 performs the processing of the neural network to which the second estimation parameter is applied, using the image captured in S910 as an input. Therefore, the neural network processing using the second estimation parameter is performed as the processing for image processing control (S1013). In the case where the CPU 101 performs the processing of S1013 on all still images temporarily stored in the memory 102 or the recording medium 120, the processing ends.

[0099] Next, refer to Fig.11To illustrate the motion image shooting process of the second embodiment. CPU 101 notifies the neural core 140 of the first inference parameter, and notifies the neural core 150 of the second inference parameter. The neural core 140 corresponds to one or more processing units, and the neural core 150 corresponds to other processing units. The neural core 140 reads the first inference parameter (S1101). Thus, the first inference parameter is set in the neural core 140. In addition, the neural core 150 reads the second inference parameter (S1102). Thus, the second inference parameter is set in the neural core 150. S1102 may be performed before S1101, or S1101 and S1102 may be performed simultaneously.

[0100] Based on the notification from the CPU 101 , the neural core 140 performs a neural network process (processing for shooting control) using the live view image as an input image ( S1103 ).

[0101] When the motion image shooting starts (S1104), the neural core 140 uses each frame image of the motion image as an input image to perform a neural network processing (processing used for shooting control) using the first inference parameter (S1105). In addition, the neural core 150 uses each frame image of the motion image as an input image to perform a neural network processing (processing used for image processing control) using the second inference parameter (S1106). S1106 can be performed before S1105, or S1105 and S1106 can be performed at the same time. CPU 101 determines whether there is an instruction to end shooting (S1107). When it is judged as "No" in S1107, the motion image shooting has not ended, and the process returns to S1105. When it is judged as "Yes" in S1107, the motion image shooting is ended, and the processing is ended.

[0102] In the moving image shooting process, the neural network process using the first inference parameter and the neural network process using the second inference parameter are performed in parallel by using a plurality of neural cores. Thus, in the moving image shooting process, the process for shooting control and the process for image processing control can be performed in parallel.

[0103] Other embodiments

[0104] In the above-mentioned embodiments, an example in which the camera apparatus 100 has the CPU 101 and the neural network processing unit 105 is described. There is a limit to the circuit scale of the neural core that can be installed on the camera apparatus 100. Therefore, in the camera apparatus 100, the neural core with a small amount of hardware resources can be efficiently utilized by performing the control of the above-mentioned embodiments. On the other hand, the above-mentioned embodiments can be applied not only to the camera apparatus 100 but also to a predetermined information processing apparatus.

[0105] For example, an edge computer can be applied as a predetermined information processing device. In this case, the edge computer has the functions of a CPU 101 and a neural network processing unit 105. The camera device 100 sends the captured image to the edge computer via the communication unit 118. The edge computer performs the processing of the above-mentioned embodiments and sends the processing results to the camera device 100. Then, the processing results are recorded in the recording medium 120. In this case, the information processing device has the functions of an acquisition unit and an output unit. In the case where the camera device 100 and the edge computer are capable of high-speed communication, the processing of each embodiment can be performed by the edge computer. In addition, the processing of each embodiment can be performed by a cloud server, etc. instead of an edge computer. Alternatively, the processing of each embodiment can be performed by an intelligent device such as a smart phone.

[0106] In addition, if Figure 1 As shown, the camera apparatus 100 can communicate with the external device 217. Therefore, part of the processing of each embodiment can be performed by the external device 217. In the case where the camera apparatus 100 has the neural network processing unit 105, each inference parameter can be acquired by the camera apparatus 100 from the external device 217 as needed.

[0107] Other embodiments

[0108] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0109] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications, equivalent structures and functions.

[0110] This application claims the benefit of Japanese Patent Application No. 2019-138934, filed Jul. 29, 2019, which is hereby incorporated by reference herein in its entirety.

Claims

1. An information processing device, comprising: A first processing unit, for performing arithmetic processing of the neural network; a second processing unit for performing arithmetic processing of the neural network; as well as a control unit for setting an inference parameter used in the arithmetic processing of the neural network performed by the first processing unit and an inference parameter used in the arithmetic processing of the neural network performed by the second processing unit, Wherein, when a predetermined condition is satisfied, the control unit changes one of the inference parameters used for arithmetic processing of the neural network performed by the first processing unit and the inference parameters used for arithmetic processing of the neural network performed by the second processing unit and does not change the other.

2. The information processing device according to claim 1, wherein: After determining a focus object at the time of shooting, the control unit changes one of an inference parameter used for arithmetic processing of a neural network performed by the first processing unit and an inference parameter used for arithmetic processing of a neural network performed by the second processing unit without changing the other.

3. The information processing device according to claim 1, wherein: When the predetermined condition is satisfied, the control unit sets the inference parameter to be applied to the processing used for image processing control to one of the inference parameter used for the arithmetic processing of the neural network performed by the first processing unit and the inference parameter used for the arithmetic processing of the neural network performed by the second processing unit.

4. The information processing device according to claim 3, wherein: The control unit sets an inference parameter to be applied to a process for shooting control to the other of an inference parameter used for arithmetic processing of a neural network performed by the first processing unit and an inference parameter used for arithmetic processing of a neural network performed by the second processing unit.

5. The information processing device according to claim 4, wherein: Before the predetermined condition is satisfied, the control unit sets the inference parameter to be applied to the processing used for shooting control to the other of the inference parameter used for the arithmetic processing of the neural network performed by the first processing unit and the inference parameter used for the arithmetic processing of the neural network performed by the second processing unit.

6. A control method for an information processing device, the information processing device having a first processing unit for performing arithmetic processing of a neural network and a second processing unit for performing arithmetic processing of a neural network, the control method comprising: When a predetermined condition is satisfied, one of an inference parameter used for arithmetic processing of the neural network performed by the first processing unit and an inference parameter used for arithmetic processing of the neural network performed by the second processing unit is changed and the other is not changed. 7 . A non-transitory computer-readable storage medium storing a computer-executable program, wherein the computer-executable program executes the control method according to claim 6 .

Images